Defense Management
DOD Needs to Demonstrate That Performance-Based Logistics Contracts Are Achieving Expected Benefits Gao ID: GAO-05-966 September 9, 2005The Department of Defense (DOD) contracts with private sector companies to perform depot maintenance of weapon systems using performance-based logistics--that is, purchasing a defined level of performance over a defined time period at a fixed cost to the government. After implementing such contracts, program offices are to validate their efficacy using cost and performance data; DOD cannot otherwise ensure cost savings and improved performance are being achieved through the use of performance-based logistics. GAO was asked to review the implementation of performance-based logistics to determine whether DOD could demonstrate cost savings and improved responsiveness from these arrangements. In conducting its review, GAO analyzed the implementation of performance-based logistics arrangements for 15 weapon system programs.
DOD program offices could not demonstrate that they have achieved cost savings or performance improvements through the use of performance-based logistics arrangements. Although DOD guidance on implementing these arrangements states program offices should update their business case analysis based on actual cost and performance data, only 1 of the 15 program offices included in GAO's review had performed such an update consistent with DOD guidance. In the single case where the program office had updated its business case analysis, it determined that the performance-based logistics contract did not result in expected cost savings and the weapon system did not meet established performance requirements. In general, program offices had not updated their business case analysis after entering into a performance-based logistics contract because they assumed that the costs for weapon system maintenance incurred under a fixed-price performance-based logistics contract would always be lower than costs under a more traditional contracting approach and because they lacked reliable cost and performance data needed to validate assumptions used. Furthermore, the Office of the Secretary of Defense has not established procedures to monitor program offices to ensure they follow guidance and update the business case analysis. Additionally, program officials said because of limitations in their own information systems, they typically relied on cost and performance data generated by the contractors' information systems to monitor performance-based logistics contracts. The program offices, however, had not determined whether contractor-provided data were sufficiently reliable to update their business case analysis. Although the Defense Contract Management Agency and the Defense Contract Audit Agency are most commonly used to monitor higher risk contracts, such as cost plus contracts, they are potential resources available to assist program offices in monitoring fixed-price performance-based contracts. In doing so, these DOD agencies have the capability to verify the reliability of contractors' information systems and collect cost and performance data needed to update their business case analysis. Until program offices follow DOD's guidance and update their business case analysis based on reliable cost and performance data, DOD cannot evaluate the extent to which performance-based logistics arrangements are achieving expected benefits and being effectively implemented within DOD.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-05-966, Defense Management: DOD Needs to Demonstrate That Performance-Based Logistics Contracts Are Achieving Expected Benefits
This is the accessible text file for GAO report number GAO-05-966
entitled 'Defense Management: DOD Needs to Demonstrate That Performance-
Based Logistics Contracts Are Achieving Expected Benefits' which was
released on September 9, 2005.
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Subcommittee on Readiness and Management Support,
Committee on Armed Services, U.S. Senate:
September 2005:
Defense Management:
DOD Needs to Demonstrate That Performance-Based Logistics Contracts Are
Achieving Expected Benefits:
[Hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-05-966]:
GAO Highlights:
Highlights of GAO-05-966, a report to the Subcommittee on Readiness and
Management Support, Committee on Armed Services, U.S. Senate:
Why GAO Did This Study:
The Department of Defense (DOD) contracts with private sector companies
to perform depot maintenance of weapon systems using performance-based
logistics ”that is, purchasing a defined level of performance over a
defined time period at a fixed cost to the government. After
implementing such contracts, program offices are to validate their
efficacy using cost and performance data; DOD cannot otherwise ensure
cost savings and improved performance are being achieved through the
use of performance-based logistics. GAO was asked to review the
implementation of performance-based logistics to determine whether DOD
could demonstrate cost savings and improved responsiveness from these
arrangements. In conducting its review, GAO analyzed the implementation
of performance-based logistics arrangements for 15 weapon system
programs.
What GAO Found:
DOD program offices could not demonstrate that they have achieved cost
savings or performance improvements through the use of performance-
based logistics arrangements. Although DOD guidance on implementing
these arrangements states program offices should update their business
case analysis based on actual cost and performance data, only 1 of the
15 program offices included in GAO‘s review had performed such an
update consistent with DOD guidance. In the single case where the
program office had updated its business case analysis, it determined
that the performance-based logistics contract did not result in
expected cost savings and the weapon system did not meet established
performance requirements. In general, program offices had not updated
their business case analysis after entering into a performance-based
logistics contract because they assumed that the costs for weapon
system maintenance incurred under a fixed-price performance-based
logistics contract would always be lower than costs under a more
traditional contracting approach and because they lacked reliable cost
and performance data needed to validate assumptions used. Furthermore,
the Office of the Secretary of Defense has not established procedures
to monitor program offices to ensure they follow guidance and update
the business case analysis. Additionally, program officials said
because of limitations in their own information systems, they typically
relied on cost and performance data generated by the contractors‘
information systems to monitor performance-based logistics contracts.
The program offices, however, had not determined whether contractor-
provided data were sufficiently reliable to update their business case
analysis. Although the Defense Contract Management Agency and the
Defense Contract Audit Agency are most commonly used to monitor higher
risk contracts, such as cost plus contracts, they are potential
resources available to assist program offices in monitoring fixed-price
performance-based contracts. In doing so, these DOD agencies have the
capability to verify the reliability of contractors‘ information
systems and collect cost and performance data needed to update their
business case analysis. Until program offices follow DOD‘s guidance and
update their business case analysis based on reliable cost and
performance data, DOD cannot evaluate the extent to which performance-
based logistics arrangements are achieving expected benefits and being
effectively implemented within DOD.
What GAO Recommends:
GAO recommended that DOD, to demonstrate whether performance-based
logistics contracts are resulting in reduced costs and increased
performance, develop procedures to track whether program offices
validate their business case decisions and verify the reliability of
contractor cost and performance data. DOD concurred with GAO‘s
recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-05-966.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact William Solis at (202)
512-8412 or solisw@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
DOD Program Offices Could Not Demonstrate Cost Savings and Performance
Improvement Resulting from the Use of Performance-Based Logistics:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Scope and Methodology:
Appendixes:
Appendix I: Comments from the Department of Defense:
Appendix II: GAO Contact and Staff Acknowledgments:
Table:
Table 1: Performance-Based Logistics Programs Included in Our Review:
Letter September 9, 2005:
The Honorable John Ensign:
Chairman:
The Honorable Daniel K. Akaka:
Ranking Minority Member:
Subcommittee on Readiness and Management Support:
Committee on Armed Services:
United States Senate:
The Department of Defense (DOD) reported that it spent $23.6 billion in
fiscal year 2004 for depot maintenance of military weapon
systems.[Footnote 1] Almost half of those expenditures were paid to
private sector companies to perform depot maintenance activities for
selected weapon systems. DOD's preferred approach for providing long-
term total system support for weapon systems is a concept it calls
performance-based logistics. Performance-based logistics is a variation
of other contractor logistics support strategies and involves defining
a level of performance that the weapon system is to achieve over a
period of time at a fixed cost to the government.[Footnote 2] Specific
performance measures are established in each contract. DOD believes
this contracting strategy for supporting weapon systems and components
will achieve higher levels of performance while minimizing costs. For
example, DOD has claimed millions of dollars in cost savings and
significant performance improvements in such areas as material
availability and logistics response time through the use of performance-
based logistics. As a result of aggressively pursuing this contracting
strategy, DOD expected an increasing number of weapon systems and
components to be maintained for the military services by private sector
companies under long-term, fixed-price, performance- based contracts.
To justify the use of a performance-based logistics strategy, according
to DOD guidance, program offices should conduct a business case
analysis prior to adopting this approach. Following implementation of a
performance-based logistics contract, program offices should update
their business case analysis to validate the assumptions used based on
actual cost and performance data and to support future weapon systems
plans.
As requested, we reviewed DOD's implementation of performance-based
logistics arrangements. Our objective was to determine whether DOD
could demonstrate cost savings or improved responsiveness through the
use of performance-based logistics arrangements. This is the second of
two reports addressing DOD's implementation of performance-based
logistics. In an August 2004 report,[Footnote 3] we identified the
types of contractor logistics support arrangements the private sector
used when outsourcing the maintenance of complex and costly equipment
with life-cycle management issues similar to those of military systems.
We also identified several lessons learned from comparisons between
private sector contractor support practices and the performance-based
logistics approach that DOD was urging the services to use. That report
recommended that DOD incorporate into guidance the private sector's
practice of using performance-based contracts to achieve economies at
the component level, rather than as a preferred practice at the
platform level. DOD issued policy memoranda and revised its guidebook
to implement our recommendation.
As part of our current review, we collected and analyzed data on 15
weapon system programs that were among those identified by the Office
of the Secretary of Defense and the military services as programs that
have successfully used performance-based logistics arrangements. We
selected weapon system programs that were sufficiently mature to
provide lessons learned on the use of performance-based logistics. We
reviewed contracts and related files and interviewed program officials
to determine how these arrangements were structured and managed. We
examined business case analysis documents prepared by the program
offices to justify the use of a performance-based logistics strategy.
We discussed with program officials the systems they used to monitor
contractor cost and performance. We also reviewed DOD policies,
procedures, and guidance related to the use of performance-based
logistics for weapon system support. We determined that the data used
were sufficiently reliable for our purposes. We performed our work from
August 2004 through June 2005 in accordance with generally accepted
government auditing standards. The Scope and Methodology section
contains more detailed information about the work we performed.
Results in Brief:
DOD program offices could not demonstrate that they have achieved cost
savings or performance improvements through the use of performance-
based logistics arrangements. Although DOD guidance on implementing
these arrangements states program offices should update their business
case analysis based on actual cost and performance data, only 1 of the
15 program offices included in our review had performed such an update
consistent with DOD guidance. In the single case where the program
office had updated its business case analysis, it determined that the
performance-based logistics contract did not result in expected cost
savings and the weapon system did not meet established performance
requirements. In general, program offices had not updated their
business case analysis after entering into a performance-based
logistics contract because they assumed that the costs for weapon
system maintenance incurred under a fixed-price, performance-based
logistics contract would always be lower than costs under a more
traditional contracting approach and because they lacked reliable cost
and performance data needed to validate assumptions used. Furthermore,
the Office of the Secretary of Defense has not established procedures
to monitor program offices to ensure they follow guidance and update
the business case analysis. Additionally, program officials said
because of limitations in their own information systems, they typically
relied on cost and performance data generated by the contractors'
information systems to monitor performance-based logistics contracts.
The program offices, however, had not determined whether the contractor-
provided data were sufficiently reliable to update their business case
analysis. Although the Defense Contract Management Agency (DCMA) and
the Defense Contract Audit Agency (DCAA) are most commonly used to
monitor higher risk contracts, such as cost plus contracts, they are
potential resources available to assist program offices in monitoring
fixed-price performance-based contracts. In doing so, these DOD
agencies have the capability to verify the reliability of contractors'
information systems and collect cost and performance data needed to
update the business case analysis. Until program offices follow DOD's
guidance and update their business case analysis based on reliable cost
and performance data, DOD cannot evaluate the extent to which
performance-based logistics arrangements are achieving expected
benefits and being effectively implemented within DOD.
This report contains recommendations aimed at ensuring that DOD program
offices follow DOD guidance for updating their business case analyses
and verify the reliability of contractor cost and performance data. In
commenting on a draft of this report, DOD concurred with our findings
and recommendations. DOD's response is included in appendix I.
Background:
DOD has increasingly relied on contractors to provide logistics support
for weapon system maintenance. These logistics support arrangements
have taken various forms. In fiscal year 1998, DOD directed the armed
services to pursue logistics support "reengineering" efforts with
contractors to achieve cost savings and improve efficiency.[Footnote 4]
A 1999 DOD study identified 30 pilot programs to test logistics support
concepts that placed greater reliance on the private sector. Some of
the pilot programs involved performance-type arrangements that were
subsequently converted to, or designated as, performance-based
logistics contracts.[Footnote 5] DOD's Quadrennial Defense Review
Report[Footnote 6] advocated the implementation of performance-based
logistics, with appropriate metrics, to compress the supply
chain[Footnote 7] by removing steps in the warehousing, distribution,
and order fulfillment processes; reducing inventories; and reducing
overhead costs while improving the readiness of major weapon systems
and commodities.
Over the last few years, DOD has issued guidance on the implementation
of performance-based logistics. In November 2001, the Office of the
Deputy Under Secretary of Defense issued guidance recommending that
program managers conduct a sound business case analysis to decide
whether they should implement performance-based logistics for new
systems and major acquisitions for already fielded systems.[Footnote 8]
In an August 2003 memorandum to the military departments, the Under
Secretary of Defense (Acquisition, Technology and Logistics) stated
that DOD should continue to increase its use of performance-based
logistics acquisitions. On February 4, 2004, the Deputy Secretary of
Defense (1) directed the Under Secretary of Defense (Acquisition,
Technology and Logistics), in conjunction with the Under Secretary of
Defense (Comptroller), to issue clear guidance on purchasing logistics
support using performance criteria and (2) directed each service to
provide a plan to aggressively implement performance-based logistics
for current and planned weapon system platforms. Then, based on
recommendations in our August 2004 report,[Footnote 9] the Under
Secretary of Defense (Acquisition, Technology and Logistics) issued a
memorandum reemphasizing that the use of this type of support strategy
was intended to optimize weapon system availability while minimizing
costs and the logistics footprint[Footnote 10] and may be applied to
weapon systems, subsystems, and components. The memorandum also
provided specific definitions of performance metrics to be used.
DOD describes performance-based logistics as the process of (1)
identifying a level of performance required by the warfighter and (2)
negotiating a performance-based arrangement between the government and
a contractor or government facility to provide long-term total system
support for a weapon system at a fixed level of annual funding. Instead
of buying spare parts, repairs, tools, and data in individual
transactions, DOD program offices that use a performance-based
logistics arrangement buy a predetermined level of performance that
meets the warfighter's objectives. Although established performance
measures should be tailored to reflect the unique circumstances of each
performance-based logistics arrangement, the measures are expected to
support five general objectives: (1) percentage of time that a weapon
system is available for a mission (operational availability); (2)
percentage of mission objectives met (operational reliability); (3)
operating costs divided by a specified unit of measure (cost per unit
usage); (4) size or presence of support required to deploy, sustain, or
move a weapon system (logistics footprint); and (5) period of time that
is acceptable between the demand or request for support and the
satisfactory fulfillment of that request (logistics response time).
Currently, a DOD task force is refining these objectives into DOD
standard performance definitions to be used by program offices in every
service when preparing performance-based logistics arrangements.
DOD guidance recommends that program offices prepare a business case
analysis prior to adopting a performance-based logistics approach to
support a weapon system.[Footnote 11] The aim of the business case
analysis is to justify the decision to enter into a performance-based
logistics contract. The business case analysis is to include cost
savings that are projected as a result of using a performance-based
logistics approach and the assumptions used in developing the business
case analysis. Furthermore, DOD guidance states that program offices
should update their business case analyses at appropriate decision
points when sufficient cost and performance data have been collected to
validate the assumptions used in developing the business case analyses,
including the costs of alternative approaches, projected cost savings,
and expected performance levels. Further, GAO Internal Control
Standards state that it is necessary to periodically review and
validate the propriety and integrity of program performance measures
and indicators. Also, actual performance data should be continually
compared against expected or planned goals, and any difference should
be analyzed. Additionally, management should have a monitoring strategy
that emphasizes to program managers their responsibility for internal
controls (i.e., to review and validate performance measures and
indicators) and that includes a plan for periodic evaluation of control
activities.
DOD Program Offices Could Not Demonstrate Cost Savings and Performance
Improvement Resulting from the Use of Performance-Based Logistics:
DOD program offices could not demonstrate that their use of performance-
based logistics arrangements had achieved cost savings and performance
improvements because they had not updated their business case analysis
as suggested by DOD guidance. Specifically, of the 15 DOD program
offices, only 1 updated its business case analysis to validate
assumptions concerning cost and performance. Other DOD program offices
had not updated their business case analysis in part because they
lacked reliable contractor cost and performance data. The program
offices typically relied on cost and performance data generated by
contractors' information systems without verifying that the data were
sufficiently reliable to update the business case analysis. Two DOD
agencies, DCMA and DCAA, have the capability to assist program offices
in monitoring fixed-price performance-based contracts, verifying the
reliability of contractors' information systems, and collecting cost
and performance data.
Projected Cost Savings and Performance Improvements Were Not Validated:
None of the 15 program offices included in our review could demonstrate
that use of a performance-based logistics arrangement had achieved cost
savings and performance improvements. Although an updated business case
analysis based on actual cost and performance data might show that cost
savings and performance improvements were being achieved, only 1 of the
15 program offices had updated its business case analysis consistent
with DOD guidance. Of the 15 program offices, 11 had developed a
business case analysis prior to entering into a performance-based
logistics arrangement.[Footnote 12] In their analysis, these program
offices projected that they would achieve significant cost savings. For
example, an Army program office projected total cost savings of $508.5
million, and a Navy program office projected cost savings of $29.7
million. However, only the Navy's T-45 program office had subsequently
updated its business case analysis consistent with DOD guidance to
determine whether cost savings were being achieved.[Footnote 13]
Realizing that the contractor was not meeting the aircraft availability
performance measure, the program office reassessed its business case
assumptions and found that costs per flying hour were higher than
estimated because the aircraft was flying fewer hours than forecasted.
As a result, the program office negotiated separate contracts for the
airframes and engines, which resulted in estimated cost savings of $144
million over 5 years.
Performance indicators tracked by the program offices showed that the
contractors met or exceeded performance requirements. Of the 15
programs, 10 reported that performance levels exceeded contract
requirements, and 5 reported that performance levels were meeting
contract requirements. For example, an Army program office reported a
weapon system availability rate of 99 percent, which is 7 percent
higher than what was projected in the business case analysis.
Similarly, a Navy program office reported a weapon system availability
rate of 97 percent, which is 7 percent higher than projected.
Despite the reported performance improvements, the program offices had
not analyzed the performance data to validate the improvements and
determine whether these improvements could be attributed directly to
their use of performance-based logistics arrangements to support the
weapon systems. In addition, we noted that program offices in the past
reported they had also met or exceeded required levels of performance
using other contractual arrangements for weapon system maintenance.
Moreover, the DOD program offices reporting that performance levels
were exceeding contract requirements under performance-based logistics
arrangements had not determined the incremental costs associated with
achieving these higher levels of performance. As a result, they had no
way of knowing whether incremental costs outweighed the benefits
derived from achieving performance levels in excess of requirements.
Program officials did not follow DOD guidance to update and validate
their business case analyses because they assumed that costs incurred
under fixed-price performance-based logistics arrangements would always
be lower than costs incurred under more traditional contracting
arrangements, and several program officials cited a lack of reliable
data needed to validate expected costs savings and improved
performance. However, the experience of the T-45 program showed that it
is possible for program offices to validate the assumptions in the
business case analysis and to determine whether expected cost savings
and performance improvements were achieved. There are also other
benefits derived from validating the assumptions used in the business
case analysis. Validation can provide a better understanding of costs
associated with the repair and maintenance of weapon systems, ensure
that proper performance metrics are in place to satisfy logistical
demand, isolate incremental costs associated with achieving higher
levels of performance, and make cost and performance data available for
contract renegotiations in order to obtain the best value for the
government. Furthermore, we did not find evidence that the Office of
the Secretary of Defense had established procedures to monitor whether
program offices were following its guidance to update their business
case analyses. The results of these updates could be used by DOD to
assess the implementation of performance-based logistics arrangements
and evaluate the extent to which performance-based logistics
arrangements are achieving expected benefits.
DOD Program Offices Relied on Contractors' Data Without Verifying the
Reliability of the Data:
DOD program offices included in our review stated that because of
limitations in their own information systems, they typically relied on
cost and performance data generated by the contractors' information
systems to monitor performance-based logistics contracts. Program
offices acknowledged limitations in their own information systems in
providing reliable data to closely monitor contractor cost and
performance. Existing systems are capable of collecting some cost and
performance information on performance-based logistics contracts;
however, according to program officials, the systems are not capturing
sufficiently detailed cost and performance information for monitoring
performance-based logistics contracts.[Footnote 14] Program officials
told us they had more confidence in the accuracy and completeness of
contractor systems than in their legacy systems. The program offices,
however, had not determined whether the contractor-provided data were
sufficiently reliable to update their business case analyses. As a
result, the program offices did not have the reliable data they needed
to validate the assumptions used in the business case analysis and to
determine whether their performance-based logistics arrangements were
achieving expected cost savings and improved performance.
As we noted in a prior report on DOD's management of depot maintenance
contracting,[Footnote 15] to reduce personnel and save costs, DOD
decided to rely more on contractors to manage and oversee fixed-price
contracts because these contracts are considered low risk. The
contractor assumes most of the risks for fixed-priced contracts, with
the government taking a more limited role in monitoring these
contracts. In our prior work on defense contract management, we
discussed the importance of monitoring contractors' systems to ensure
the accuracy and completeness of information generated by these
systems.[Footnote 16] In addition, during our review of the private
sector's use of performance-based logistics,[Footnote 17] we noted that
private-sector companies that use performance-based logistics
contracts, whether fixed price or cost-plus, closely monitor cost and
performance information to effectively manage their contracts. These
companies said they rely on their own systems and personnel to verify
the cost and quality of work performed by the contractor. The private
sector takes this approach (1) to ensure that expected costs under the
contracts are accurate and meet the company's reliability standards,
(2) to validate the business case decision used to justify a
performance-based logistics arrangement, and (3) to obtain the data
necessary to renegotiate the contract.
DOD Program Offices Made Limited Use of Resources Available to Assist
in Monitoring Performance-Based Logistics Contracts:
DCMA and DCAA have the capability to monitor contractor cost and
performance, verify the reliability of contractor-provided data, and
collect detailed cost and performance data. However, most of the DOD
program offices we reviewed made limited use of these agencies'
resources because they viewed fixed-price performance-based logistics
contracts to be low risk compared with other types of contracts. Before
a contract is awarded, DCMA can provide advice and service to help
construct effective solicitations, identify potential risk, select the
most capable contractors, and write contracts that meet the needs of
DOD customers. After the contract is awarded, DCMA can monitor
contractors' information systems to ensure that cost, performance, and
delivery schedules are in compliance with the terms and conditions of
the contracts. DCAA performs contract audits for DOD components and
provides accounting and financial advisory services during contract
negotiation and administration of contracts.
DCMA and DCAA officials said that they have a greater role in
monitoring cost information for cost-plus contracts because such
contracts are considered high risk. According to DCMA and DCAA
officials, their level of oversight is significantly less for fixed-
priced contracts, including performance-based logistics arrangements,
because DOD considers these contracts to be low risk, thereby
diminishing the need for monitoring contractor performance. Without a
request from program offices or specific contract clauses, DCMA and
DCAA generally would not conduct periodic reviews or audits of fixed-
price contracts to verify cost and performance information. DCMA and
DCAA officials also said that in the past, monitoring fixed-price
contracts was included in their workload, but because of a reduction in
staff and streamlining of operations, they focused their efforts on
contract areas that have the highest risk for cost growth. DCMA and
DCAA officials said they would support increasing their role in
monitoring fixed-price performance-based contracts depending on the
availability of their resources.
Conclusions:
DOD is expanding its use of performance-based logistics as its
preferred support strategy in support of weapon systems but has not yet
demonstrated that this long-term support strategy is being effectively
implemented DOD-wide. DOD guidance states that program offices, after
entering into performance-based logistics arrangements, should update
their original business case analysis using actual cost and performance
data to validate their assumptions, but most of the program offices we
reviewed had not followed this guidance, and the Office of the
Secretary of Defense was not monitoring whether program offices were
following the guidance. The program offices therefore could not
substantiate that cost savings and performance improvements for weapon
system support were being achieved through the use of performance-based
logistics arrangements. Program offices also have lacked reliable cost
and performance data needed to validate the results of performance-
based logistics arrangements. Reliable data could be collected and
analyzed by increasing oversight of these contracts with the assistance
of DCMA and DCAA.
Recommendations for Executive Action:
To demonstrate that performance-based logistics arrangements are
resulting in reduced costs and increased performance, and to improve
oversight of performance-based logistics contracts, we recommend that
the Secretary of Defense direct the Under Secretary of Defense
(Acquisition, Technology and Logistics) to take the following two
actions:
1. Reaffirm DOD guidance that program offices update their business
case analyses following implementation of a performance-based logistics
arrangement and develop procedures, in conjunction with the military
services, to track whether program offices that enter into these
arrangements validate their business case decisions consistent with DOD
guidance.
2. Direct program offices to improve their monitoring of performance-
based logistics arrangements by verifying the reliability of contractor
cost and performance data. The program offices may wish to increase the
role of DCMA and DCAA in overseeing performance-based logistics
contracts.
Agency Comments and Our Evaluation:
In commenting on a draft of this report, DOD concurred with our
recommendations regarding the validation of business case decisions for
performance-based logistics arrangements and verification of
reliability of contractor data. While DOD was generally responsive to
our recommendations, specific details on how DOD planned to validate
and verify contractor data were not provided.
Regarding our recommendation to reaffirm guidance and develop
procedures to track whether program offices validate their business
case decisions, DOD stated that the department will reaffirm DOD
guidance on updating the business case analysis after implementing
performance-based logistics arrangements and will work with the
military services to develop procedures to track whether program
offices validate their business case decisions consistent with DOD
guidance.
With regard to our second recommendation to direct program offices to
verify the reliability of contractor cost and performance data, DOD
stated that it will issue guidance on verifying the reliability of
contractor cost and performance data. DOD did not provide specific
information on what the guidance would include nor did it indicate
whether it would increase the use of DCMA or DCAA to verify the
reliability of contractor cost and performance data.
DOD also provided technical comments, which we have incorported as
appropriate.
Scope and Methodology:
To determine whether DOD could demonstrate that cost savings and
improved performance were being achieved through the use of performance-
based logistics arrangements, we collected and analyzed data on 15
weapon system programs identified by the Office of the Secretary of
Defense and the military services as programs that have successfully
used performance-based logistics arrangements. The 15 programs are
listed in table 1.
Table 1: Performance-Based Logistics Programs Included in Our Review:
Military service: Air Force;
Weapon system or component: C-17;
Program office: Air Force Materiel Command.
Military service: Air Force;
Weapon system or component: F-117;
Program office: Air Force Materiel Command.
Military service: Air Force;
Weapon system or component: JSTARS;
Program office: Warner Robins Air Logistics Center.
Military service: Air Force;
Weapon system or component: C-130J;
Program office: Air Force Materiel Command.
Military service: Navy;
Weapon system or component: ALR-67 (V3);
Program office: Naval Supply Systems Command.
Military service: Navy;
Weapon system or component: Auxiliary Power Units;
Program office: Naval Supply Systems Command and Naval Air Systems
Command.
Military service: Navy;
Weapon system or component: F-18 E/F FIRST;
Program office: Naval Supply Systems Command and Naval Air Systems
Command.
Military service: Navy;
Weapon system or component: F-404;
Program office: Naval Supply Systems Command and Naval Air Systems
Command.
Military service: Navy;
Weapon system or component: T-45 engines;
Program office: Naval Supply Systems Command and Naval Air Systems
Command.
Military service: Navy;
Weapon system or component: V-22 engines [A];
Program office: Naval Air Systems Command.
Military service: Navy/Marine Corps;
Weapon system or component: KC- 130J [B];
Program office: Naval Supply Systems Command and Naval Air Systems
Command.
Military service: Army;
Weapon system or component: HIMARS;
Program office: Aviation and Missile Command.
Military service: Army;
Weapon system or component: Javelin CLU;
Program office: Aviation and Missile Command.
Military service: Army;
Weapon system or component: TOW-ITAS;
Program office: Aviation and Missile Command.
Military service: Army;
Weapon system or component: TUAV Shadow;
Program office: Aviation and Missile Command.
Source: GAO analysis of services' data.
[A] This is a joint contract with the Navy and Marine Corps.
[B] The Marine Corps' maintenance contracting was handled by a Navy
team.
[End of table]
We reviewed DOD and service policies, procedures, and guidance related
to the use of performance-based logistics and met with program
officials to discuss how their performance-based logistics contracts
were structured and managed and how these contracts were validated to
ensure that cost savings and improved performance were being achieved
as a result of using performance-based logistics. We also obtained and
analyzed available documentation, including business case analyses,
contracts, and related files. We did not assess the methodology program
offices used to prepare their business case analyses or the quality of
these analyses. We discussed with program officials the systems they
used to monitor contractor cost and performance. We also interviewed
officials from the Office of the Secretary of Defense and military
department headquarters to discuss implementation of performance-based
logistics, lessons learned, and the benefits derived from using
performance-based logistics approaches and practices. To determine how
private-sector companies ensure that cost and performance levels under
a performance-based contract are as expected, we reviewed the
information provided by seven companies identified in our prior report
that used complex and costly equipment that had life-cycle management
issues similar to military weapon systems, and outsourced some portion
of their maintenance work under performance-based contracts. These
seven companies consisted of six airline companies and one mining
company. We contacted officials at DCMA and DCAA to determine those
agencies' roles in monitoring the costs and performance of fixed-priced
contracts, including performance-based logistics contracts, how audits
are requested or initiated, and the procedures for reporting the
results of the audits.
We are sending this report to the Chairman and Ranking Minority Member,
Senate Subcommittee on Readiness and Management Support, Committee on
Armed Services. We will also send copies to the Under Secretary of
Defense (Acquisition, Technology and Logistics). Copies of this report
will be made available to others upon request. In addition, the report
will be available at no charge on GAO's Web site at [Hyperlink,
http://www.gao.gov]. If you or your staff have any questions on the
matters discussed in this report, please contact me at (202) 512-8412
or [Hyperlink, solisw@gao.gov]. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last
page of this report. GAO staff who made major contributions to this
report are listed in appendix II.
Signed by:
William M. Solis:
Director, Defense Capabilities and Management:
[End of section]
Appendixes:
Appendix I: Comments from the Department of Defense:
DEPUTY UNDER SECRETARY OF DEFENSE FOR LOGISTICS AND MATERIEL READINESS:
3500 DEFENSE PENTAGON:
WASHINGTON, DC 20301-3500:
August 30, 2005:
Mr. William M. Solis:
Director, Defense Capabilities and Management:
U.S. Government Accountability Office:
441 G Street, N.W.:
Washington, DC 20548:
Dear Mr. Solis:
This is the Department of Defense (DoD) response to the GAO draft
report, "DEFENSE MANAGEMENT: DoD Needs to Demonstrate That Performance-
Based Logistics Contracts Are Achieving Expected Benefits," dated
August 16, 2005, (GAO Code 350551/GAO-05-966).
The Department generally concurs with the report and concurs with the
recommendations. Detailed comments on the GAO recommendations are
provided in the attachment. The Department appreciates the opportunity
to comment on the draft report.
Sincerely,
Signed for:
Jack Bell:
GAO DRAFT REPORT - DATED August 16, 2005 GAO CODE 350551/GAO-05-966:
"DEFENSE MANAGEMENT: DOD Needs to Demonstrate That Performance-Based
Logistics Contracts Are Achieving Expected Benefits"
DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
direct the Under Secretary of Defense (Acquisition, Technology and
Logistics) to reaffirm DoD guidance that program offices update their
business case analysis following implementation of a performance-based
logistics arrangement and develop procedures, in conjunction with the
military services, to track whether program offices that enter into
these arrangements validate their business case decisions consistent
with DoD guidance. (Pages 9 and 10/GAO Draft Report):
DOD RESPONSE: Concur. The Office of the Under Secretary of Defense
(Acquisition, Technology and Logistics) will reaffirm DoD guidance
relating to updating of the business case analysis following
implementation of a performance based logistics arrangement as well as
work with the military departments to develop an appropriate method to
track whether program offices validate their business case decisions
consistent with DoD guidance.
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
direct the Under Secretary of Defense (Acquisition, Technology and
Logistics) to direct program offices to improve their monitoring of
performance-based logistics arrangements by verifying the reliability
of contractor cost and performance data. The program offices may wish
to increase the role of DCMA and DCAA in overseeing performance-based
logistics contracts.
(Page 10/GAO Draft Report):
DOD RESPONSE: Concur. The Office of the Under Secretary of Defense
(Acquisition, Technology and Logistics) will issue guidance on
verifying the reliability of contractor cost and performance data.
[End of section]
Appendix II: GAO Contact and Staff Acknowledgments:
GAO Contact:
William M. Solis (202) 512-8412 or s [Hyperlink, solisw@gao.gov]
olisw@gao.gov.
Acknowledgments:
In addition to the contact named above, Thomas Gosling, Assistant
Director; Thom Barger; Judith Collins; Pamela Valentine; and Cheryl
Weissman were major contributors to this report.
(350551):
FOOTNOTES
[1] Depot maintenance encompasses materiel maintenance or repair
requiring the overhauling, upgrading, or rebuilding of parts,
assemblies, or subassemblies and the testing and reclamation of
equipment.
[2] Performance-based logistics arrangements for weapon system support
can include either a contract with a private sector company or an
arrangement to use a government facility.
[3] GAO, Defense Management: Opportunities to Enhance the
Implementation of Performance-Based Logistics, GAO-04-715 (Washington,
D.C.: Aug. 16, 2004).
[4] Reengineering was the term DOD used to describe its efforts to
increase the cost-effectiveness of its weapon systems throughout their
life cycles by ensuring that support infrastructures were competitive,
timely, and efficient.
[5] Examples of these logistics support arrangements include (1)
contractor logistics support, where the contractor provided most or all
elements of logistics support, including depot maintenance; (2) total
system performance responsibility, under which the contractor assumed
responsibility for the weapon system's life-cycle management; and (3)
total system sustainment responsibility or total system support
responsibility, which gave the contractor responsibility for all
contracted sustainment actions including parts management and depot-
level repair.
[6] DOD, Quadrennial Defense Review Report, (Washington, D.C.: Sept.
30, 2001).
[7] Supply chain management refers to all of the interrelated
components and processes required to ensure that the correct amount of
product is in the correct location at the right time and at the lowest
cost.
[8] DOD, Product Support for the 21ST Century: A Program Manager's
Guide to Buying Performance (Washington, D.C.: Nov. 6, 2001).
[9] GAO-04-715.
[10] The logistics footprint refers to the size or presence of the
government or contractor personnel, equipment, inventory, facilities,
transportation assets, or real estate needed to deploy, sustain, and
move the weapon systems or components.
[11] DOD, Performance-Based Logistics: A Program Manager's Product
Support Guide (Washington, D.C.: Sept. 30, 2004); Product Support
Strategy (PSS) Business Case Analysis (BCA) Guiding Principles,
(Washington, D.C.: Jan. 23, 2004); and Business Case Development Guide
(Washington, D.C.).
[12] Four of the 15 program offices we reviewed had not prepared a
business case analysis. Officials at two program offices stated that
their contracting arrangements did not meet the existing definition of
a performance-based logistics contract and, therefore, the requirement
to conduct a business case analysis was not applicable. Subsequently,
the Under Secretary of Defense (Acquisition, Technology and Logistics)
issued guidance on purchasing weapon systems logistics support using
performance-based criteria and clarified the types of contract
arrangements to be considered as performance-based logistics
arrangements. DOD determined that under these clarified criteria, these
two contracts meet the definition of a performance-based logistics
contract. The other two program offices determined that they lacked
sufficient cost and performance information necessary to prepare
business case analyses. To collect baseline cost and performance data,
they entered into cost-plus contracts rather than fixed-price
contracts. On the basis of an assessment of baseline data collected in
the course of the cost-plus contracts, the program offices expect to
determine at a later time if it would be advantageous to transition
from the cost-plus contract to fixed-price performance-based logistics
arrangements. Before doing so, they would prepare a business case
analysis as suggested under DOD guidance.
[13] The T-45 Goshawk is a tandem-seat jet trainer aircraft used by the
Navy and Marine Corps for training in carrier aviation and tactical
strike missions.
[14] In a previous report, we found that despite a significant
investment in business systems, DOD continues to have long-standing
financial and business management problems that preclude the department
from producing reliable and timely information for making decisions.
See GAO, DOD Business System Modernization: Billions Being Invested
without Adequate Oversight, GAO-05-381 (Washington, D.C.: April 29,
2005).
[15] GAO, Defense Depot Maintenance: Contracting Approaches Should
Address Workload Characteristics, GAO/NSIAD-98-130 (Washington, D.C.:
June 15, 1998).
[16] GAO, High Risk Series: Defense Contract Management, GAO/HR-97-4
(Washington, D.C.: Feb. 1997).
[17] See GAO-04-715.
GAO's Mission:
The Government Accountability Office, the investigative arm of
Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order
GAO Products" heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director,
NelliganJ@gao.gov
(202) 512-4800
U.S. Government Accountability Office,
441 G Street NW, Room 7149
Washington, D.C. 20548:
