Defense Transportation
Monitoring Costs and Benefits Needed While Implementing a New Program for Moving Household Goods Gao ID: GAO-03-367 April 18, 2003The Department of Defense (DOD) spends more than $1.7 billion each year to move and store over 600,000 household goods shipments when relocating military personnel. It conducted and evaluated several pilot program studies aimed at fixing its problem-plagued program and, in 2002, issued a report to Congress with three recommendations. The 1997 Defense Appropriations Act Conference Report directed GAO to validate the results achieved by the pilot programs. In response, GAO examined the extent to which DOD's recommendations to Congress (1) offer solutions to long-standing problems in the current program and (2) are supported by the evaluation's findings and should be implemented. GAO also assessed the soundness of methodologies used by DOD to develop cost estimates to implement the recommendations.
The recommendations in DOD's report to Congress have the potential to resolve several long-standing problems found in the current personal property program, which manages the transportation and storage of household goods. The recommendations, if implemented, would (1) reengineer the claims process to reduce the length of time it currently takes to resolve claims for lost, destroyed, or damaged household goods and increase the reimbursement rates that military personnel currently receive for their losses; (2) use performance-based service contracts to improve the generally low quality of service that DOD currently gets from the moving industry; and (3) put in place new information technology with interface capabilities to enable program managers and users to monitor in-transit shipments and track the number and cost of shipments processed each year. The recommendations in DOD's report to Congress are supported by the Transportation Command's evaluation of the pilot programs' findings and should be implemented within budget constraints. DOD's approach in conducting the evaluation was methodologically sound: It developed an evaluation plan to guide its work and adjusted the plan when necessary to address differences in the pilot programs' approaches. While the shipments included in the evaluation do not represent all shipment types managed annually by DOD, GAO believes that the evaluation results provide sufficient information to allow DOD to initiate actions to improve its current personal property program. GAO found that the soundness of methodologies used to develop DOD's cost estimates varied. Therefore, DOD's ability to implement changes to the existing program within the cost estimates DOD reported to Congress is uncertain. GAO found that the estimate to implement the information technology recommendation was $7 million rather than the $4 million to $6 million estimate DOD reported to Congress. In developing cost estimates for the remaining recommendations, DOD did not provide the same level of evidentiary support for one of the three adjustments it used to align the pilot programs' costs to current program costs. As a result, GAO questions the extent to which these recommendations can be implemented within DOD's estimated 13 percent increase over current program costs. While DOD believes it used a conservative approach in developing this 13 percent estimate, it has not quantified the risk associated with the projection, which could provide the military services and Congress information needed to develop and review future budget requests for this program. Without providing the range of possible cost increases and the risk regarding the likelihood of achieving this 13 percent projection within that range, DOD may find a repeat of what occurred during the pilots, where the military services terminated participation in one of the pilot programs due to costs exceeding projections. GAO also found that without carefully monitoring costs during the implementation phase and assessing costs and benefits from a period succeeding full implementation of the recommendations, DOD would not have the information needed to determine if anticipated improvements in the program are being achieved at a reasonable cost.
RecommendationsOur recommendations from this work are listed below with a Contact for more information. Status will change from "In process" to "Open," "Closed - implemented," or "Closed - not implemented" based on our follow up work.
Director: Team: Phone:GAO-03-367, Defense Transportation: Monitoring Costs and Benefits Needed While Implementing a New Program for Moving Household Goods
This is the accessible text file for GAO report number GAO-03-367
entitled 'Defense Transportation: Monitoring Costs and Benefits Needed
While Implementing a New Program for Moving Household Goods' which was
released on April 18, 2003.
This text file was formatted by the U.S. General Accounting Office
(GAO) to be accessible to users with visual impairments, as part of a
longer term project to improve GAO products‘ accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to Webmaster@gao.gov.
Report to Congressional Committees:
United States General Accounting Office:
GAO:
April 2003:
DEFENSE TRANSPORTATION:
Monitoring Costs and Benefits Needed While Implementing a New Program
for Moving Household Goods:
GAO-03-367:
GAO Highlights:
Highlights of GAO-03-367, a report to Congressional Committees.
Why GAO Did This Study:
The Department of Defense (DOD) spends more than $1.7 billion each year
to move and store over 600,000 household goods shipments when
relocating military personnel. It conducted and evaluated several pilot
program studies aimed at fixing its problem-plagued program and, in
2002, issued a report to Congress with three recommendations. The 1997
Defense Appropriations Act Conference Report directed GAO to validate
the results achieved by the pilot programs. In response, GAO examined
the extent to which DOD‘s recommendations to Congress (1) offer
solutions to long-standing problems in the current program and (2) are
supported by the evaluation‘s findings and should be implemented. GAO
also assessed the soundness of methodologies used by DOD to develop
cost estimates to implement the recommendations.
What GAO Found:
The recommendations in DOD‘s report to Congress have the potential to
resolve several long-standing problems found in the current personal
property program, which manages the transportation and storage of
household goods. The recommendations, if implemented, would
* reengineer the claims process to reduce the length of time it
currently takes to resolve claims for lost, destroyed, or damaged
household goods and increase the reimbursement rates that military
personnel currently receive for their losses;
* use performance-based service contracts to improve the generally low
quality of service that DOD currently gets from the moving industry;
and
* put in place new information technology with interface capabilities
to enable program managers and users to monitor in-transit shipments
and track the number and cost of shipments processed each year.
The recommendations in DOD‘s report to Congress are supported by the
Transportation Command‘s evaluation of the pilot programs‘ findings and
should be implemented within budget constraints. DOD‘s approach in
conducting the evaluation was methodologically sound: It developed an
evaluation plan to guide its work and adjusted the plan when necessary
to address differences in the pilot programs‘ approaches. While the
shipments included in the evaluation do not represent all shipment
types managed annually by DOD, GAO believes that the evaluation results
provide sufficient information to allow DOD to initiate actions to
improve its current personal property program.
GAO found that the soundness of methodologies used to develop DOD‘s
cost estimates varied. Therefore, DOD‘s ability to implement changes to
the existing program within the cost estimates DOD reported to Congress
is uncertain. GAO found that the estimated to implement the information
technology recommendation was $7 million rather than the $4 million to
$6 million estimate DOD reported to Congress. In developing cost
estimates for the remaining recommendations, DOD did not provide the
same level of evidentiary support for one of the three adjustments it
used to align the pilot programs‘ costs to current program costs. As a
result, GAO questions the extent to which these recommendations can be
implemented within DOD‘s estimated 13 percent increase over current
program costs. While DOD believes it used a conservative approach in
developing this 13 percent estimate, it has not quantified the risk
associated with the projection, which could provide the military
services and Congress information needed to develop and review future
budget requests for this program. Without providing the range of
possible cost increases and the risk regarding the likelihood of
achieving this 13 percent projection within that range, DOD may find a
repeat of what occurred during the pilots, where the military services
terminated participation in one of the pilot programs due to costs
exceeding projections.
GAO also found that without carefully monitoring costs during the
implementation phase and assessing costs and benefits from a period
succeeding full implementation of the recommendations, DOD would not
have the information needed to determine if anticipated improvements in
the program are being achieved at a reasonable cost.
What GAO Recommends:
GAO recommends that DOD implement the recommendations within budget
constraints, quantify the risk associated with achieving its cost
estimates, monitor costs during the implementation phase, and assess
the new program to determine if anticipated improvements were achieved
at a reasonable cost. DOD agreed with three recommendations, but did
not agree with the need to quantify the risk associated with achieving
its cost estimates.
www.gao.gov/cgi-bin/getrpt?GAO-03-367.
To view the full report, including the scope
and methodology, click on the link above.
For more information, contact William M. Solis at (202) 512-8365 or
solisw@gao.gov.
[End of section]
Contents:
Letter:
Results in Brief:
Background:
DOD‘s Recommendations Offer Solutions to Long-Standing Problems:
Transportation Command‘s Evaluation of Pilot Programs Supports DOD‘s
Three Recommendations:
Ability to Implement New Program within Cost Estimates Reported to
Congress Is Uncertain:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Scope and Methodology:
Appendix II: Overview of Current Personal Property Program
and Pilot Programs:
Current Program:
Military Traffic Management Command‘s Reengineered Personal Property
Program:
The Department of Defense‘s Full Service Moving Project:
Navy‘s Service Member Arranged Move Pilot Program:
Appendix III: Comments from the Department of Defense:
Related GAO Products:
Tables:
Table 1: Claims-Related Features of the Current Personal Property
Program and Pilot Programs:
Table 2: Quality of Service-Related Features of the Current Personal
Property Program and Pilot Programs:
Table 3: Data Reliability-Related Features of the Current Personal
Property Program and Pilot Programs:
This is a work of the U.S. Government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. It may contain
copyrighted graphics, images or other materials. Permission from the
copyright holder may be necessary should you wish to reproduce
copyrighted materials separately from GAO‘s product.
United States General Accounting Office:
Washington, DC 20548:
April 18, 2003:
Congressional Committees:
Military personnel and their families can expect to relocate many times
during a service member‘s career. As the moving industry‘s single
largest customer, the Department of Defense (DOD) spends more than $1.7
billion annually for its personal property program, which provides
household goods transportation and storage services for military
personnel and their families when they relocate. The program manages
more than 600,000 personal property shipments each year. DOD has
experienced long-standing problems with its current personal property
program, including excessive loss or damage to property, high claims
costs incurred by the government, and poor quality of service from
moving companies. Moreover, the program‘s data management system does
not provide reliable information on the status of individual shipments
or on the types of shipments and their costs.
In an effort to test alternative approaches and improve the quality of
its personal property program, DOD has carried out several initiatives
over the past 8 years. In 2000, the U.S. Transportation
Command[Footnote 1] began to collect data from one of the three pilot
programs to evaluate alternative approaches for improving the current
program.[Footnote 2] The Transportation Command compared the features
of the current program with those of the three pilot programs, and in
June 2002 it submitted a report to the Deputy Under Secretary of
Defense (Logistics) outlining its evaluation strategy, findings, and
recommendations. In its November 12, 2002, report to Congress, DOD
included the three recommendations resulting from the evaluation and
the estimated additional costs required to implement program
improvements based on a subsequent assessment of pilot and current
programs‘ operations.
Our involvement in this issue stems from the conference report on the
1997 Defense Appropriations Act, which directed us to review the data
collected from the pilot programs and validate the results before DOD
expands any of the pilot programs, and a subsequent request from the
Subcommittee on Readiness of the House Committee on Armed Services.
This report supplements our previous testimony[Footnote 3] on DOD‘s
efforts to enhance its current program, as well as our status
report[Footnote 4] and status briefings to Congress on DOD‘s efforts to
implement a methodologically sound evaluation of the pilot programs. As
agreed with your offices, for this report we assessed the extent to
which the recommendations in DOD‘s November 2002 report (1) offer
solutions to long-standing problems in the personal property program
and (2) are supported by the evaluation‘s findings and should be
implemented. In addition, we assessed the soundness of the methodology
used by DOD to develop the cost estimates for implementing the
recommendations.
To conduct our assessment of DOD‘s three recommendations to Congress,
we reviewed DOD and GAO prior reports on the personal property program,
interviewed DOD officials and private-sector contractors involved in
the current and pilot personal property programs and the Transportation
Command‘s evaluation, examined the methodology and findings of the
Transportation Command‘s evaluation of the three pilot programs,
reviewed the methodology and data generated by each of the three pilot
programs, and assessed the methodologies used to develop the cost
estimates for implementing DOD‘s recommendations. We did not make an
assessment of whether the anticipated benefits to be derived from
implementing the three recommendations would warrant the additional
costs DOD projects will be required to fund these improvements. The
scope and methodology we used in our review are described in further
detail in appendix I.
Results in Brief:
The three recommendations in DOD‘s report to Congress offer solutions
to several long-standing problems in the current personal property
program. Specifically, the recommendations address previously
identified problems with the liability/claims process (including the
lengthy claims process, reimbursement rates that may not fully
compensate service members for loss and damage incurred during their
moves, and high claims costs to the government), the low quality of
shipping services, and the inability to track shipments and their
costs. If implemented, DOD‘s first recommendation--to reengineer the
claims process--has the potential to reduce the length of time it
currently takes to resolve claims for lost, destroyed, or damaged
household property and increase the reimbursement rates that military
personnel currently receive for their losses. The recommendation also
has the potential to reduce the high cost of claims incurred by DOD.
The second recommendation--to use performance-based service contracts-
-has the potential to improve the generally low quality of service that
DOD currently receives from moving companies. The third recommendation-
-to put in place new information technology with interface
capabilities--has the potential to enable the personal property program
managers and other users of the program to reliably monitor in-transit
shipments and track the number and the cost of shipments processed each
year so that accurate reporting can be provided to DOD and Congress.
The recommendations in DOD‘s report to Congress are supported by the
Transportation Command‘s evaluation of the pilot programs‘ findings and
should be implemented within budget constraints. We found that the
Transportation Command‘s approach in conducting the evaluation was
methodologically sound: It developed an evaluation plan to guide its
work and adjusted the plan when necessary to address differences in the
pilot programs‘ approaches. While the shipments included in the
evaluation do not represent all shipment types managed annually by DOD,
we believe that the evaluation results provide sufficient information
to allow DOD to initiate actions to improve its current personal
property program.
Our review showed that the soundness of the methodologies used by DOD
to develop cost estimates for implementing the three recommendations
varied. Therefore, DOD‘s ability to implement changes to the existing
program within the cost estimates reported to Congress is uncertain.
Our review found that the total initial cost for implementing the
information technology improvements recommendation would more likely be
$7 million rather than DOD‘s $4 million to $6 million estimate. We
agreed that the premise of two of three adjustments DOD used to develop
the 13 percent cost increase to implement the claims process and
performance-based service contract recommendations was sound. However,
we are less assured about the extent to which the projected cost
savings related to a third adjustment may occur because the adjustment
was not supported by historical experience or by the same quality of
data provided for the other adjustments. While DOD believes it can
incorporate the three recommendations into a new program within its
proposed 13 percent increase due to the conservative approach it took
in developing this estimate, we believe that by quantifying the risk
associated with this projection, DOD could provide the military
services and Congress information needed to develop and review future
budget requests for this program. The need for this type of information
is further supported based on the long-standing problems associated
with the current program and the large increase in costs contributing
to the military services‘ decision to terminate participation in one of
the pilot programs. Further, without carefully monitoring costs during
the implementation process and assessing costs and benefits from a
period succeeding full implementation of the recommendations, DOD will
not have the information needed to determine if anticipated
improvements in the program are being achieved at a reasonable cost.
Currently, DOD is beginning planning efforts to implement the
recommendations. These efforts do not include monitoring and evaluating
costs and benefits during the implementation phase and post
implementation of the recommendations in a new program.
We are recommending that DOD initiate actions that will implement the
recommendations contained in its report to Congress within budget
constraints, quantify the risk associated with achieving its cost
estimates, monitor costs during the implementation phase to ensure that
the proposed changes are being achieved within an acceptable and a
predefined range, and assess the personal property program after the
recommendations have been implemented to determine whether anticipated
improvements are being achieved at a reasonable cost.
In commenting on a draft of this report, DOD concurred with three of
our four recommendations, including initiating actions to implement the
recommendations contained in its report to Congress, monitoring costs
during the implementation phase, and assessing the program after DOD‘s
recommendations have been implemented. DOD partially concurred with the
remaining recommendation, i.e., to provide the military services and
Congress with additional information to quantify the risk associated
with achieving the projected 13 percent cost estimate. In its response,
DOD stated that the 5 percent reduction that it made to the pilot
programs‘ average costs to adjust for economies of scale/program
efficiencies was reasonable and very conservative. Further, DOD
continues to believe that the program can be implemented within the 13
percent increase and noted that one of the military services validated
this estimate. Therefore, DOD did not see the value added in providing
the military services or Congress a formal risk assessment. We find
that these statements still do not provide a basis for the 5 percent
reduction and do not indicate the level of risk associated with
implementing the recommendation within this estimate. We continue to
believe that this information needs to be developed to help the
military services prepare their budgets. The military services
terminated participation in one of the pilot programs because actual
costs were significantly greater than the projections provided to them
for budgetary purposes. Providing a measure of risk for the 13 percent
projection could help prevent a repeat of what occurred during the
pilot programs. Without this risk information, the military services
will have to wait until after the moving industry submits bids to find
out if DOD‘s projection was reliable.
Background:
DOD‘s personal property program is managed centrally by the Military
Traffic Management Command headquarters and administered locally by
about 200 military service and DOD transportation offices around the
world. The program relies on more than 1,200 domestic commercial
carriers and 150 freight forwarders for international shipments to
provide household goods transportation and storage services for
military personnel and their families when they relocate. The military
services pay shipment and storage-related costs from their military
personnel accounts and loss and damage claims and personal property
shipment office expenses through their operations and maintenance
accounts. The program has remained virtually unchanged for nearly 40
years. It involves a complex process of qualifying carriers, soliciting
rates, distributing moves, evaluating transportation providers‘
performance, paying invoices, and settling claims. Among the program‘s
many challenges is ensuring that the moving industry provides adequate
year-round capacity, especially during the summer peak-moving season
when most service members, as well as the general public, schedule
their moves.
In prior reports, both DOD and GAO have identified problems related to
the loss and damage claims process and the low quality of service from
movers. In designing and implementing its evaluation plan, the
Transportation Command also noted that weaknesses in the current
program‘s data management system precluded DOD from being able to track
shipments in transit and from being able to extract reliable data on
the number and types of shipments managed annually and their associated
costs.
In response to the long-standing problems, DOD has undertaken a number
of pilot program studies to find ways to improve the process of
shipping service members‘ household goods. In August 1996, the Deputy
Under Secretary of Defense (Logistics) tasked the Transportation
Command with evaluating alternative approaches to the current program
and recommending changes in the program based on the results of its
evaluation. The Transportation Command identified three ongoing or
planned pilot programs to include in its evaluation and began to
collect data for its analysis from one of them in 2000. These three
pilot programs shared some common features, such as testing
performance-based service contracts and providing full replacement
(rather than depreciated) value for loss or damage. Each one also had
some distinctive features, such as allowing service members to
participate in selecting their movers and contracting out installation
personal property shipment office functions to private-sector move
manager companies.
The three pilot programs are summarized below. Further information on
each program, as well as DOD‘s current personal property program, is
provided in appendix II.
* The Military Traffic Management Command‘s Reengineered Personal
Property Program operated from the military services‘ and the Coast
Guard‘s installations located in North Carolina, South Carolina, and
Florida. It used military installation personal property shipment
office personnel, as the current program does, and developed a new data
management system that tracked both the movement of individual
shipments and information on the number and cost of shipments.
* The Department of Defense‘s Full Service Moving Project operated from
the military services‘ and the Coast Guard‘s installations located in
the National Capital Region (the Washington, D.C., metropolitan area),
Georgia, and North Dakota. It contracted the management of shipments to
private-sector companies and offered optional relocation services, such
as referrals for rental assistance and purchase and sale of real estate
services, to personnel participating in the pilot program.
* The Navy‘s Service Member Arranged Move Pilot Program operated from
Navy installations located in the states of California, Connecticut,
Virginia, and Washington. It designated current staff within the
installation personal property shipment offices as ’move coordinators“
to provide assistance, allowed participants to pre-select
transportation providers, and paid for moves through government
purchase cards.
In June 2002, the Transportation Command submitted a report containing
its evaluation results and proposed three recommendations to the Deputy
Under Secretary of Defense (Logistics). After reviewing the results and
receiving comments from the military services, DOD submitted its
report, dated November 12, 2002, to Congress. DOD‘s report contained
the same three recommendations contained in the Transportation
Command‘s report. DOD also provided cost estimates for implementing the
recommendations. The three recommendations were to:
* reengineer the liability/claims process by adopting commercial
practices of minimum valuation, simplifying the filing of claims, and
providing direct settlement with the carrier;
* change the acquisition process to implement performance-based service
contracts; and:
* implement information technology improvements, which could interface
functions across such areas as personnel, transportation, financial,
and claims.
In its report to Congress, DOD estimated that reengineering the
liability/claims process and changing the acquisition process to
implement performance-based service contracts would increase the
current program‘s estimated $1.7 billion cost by 13 percent.
Implementing the information technology improvements to enhance its
data management capabilities and to provide training to users was
estimated at an additional $4 million to $6 million.
DOD also estimated that efforts to implement the changes to the current
program would be completed by the first quarter of fiscal year 2005.
DOD has developed a plan of action and milestones for designing the new
personal property program. This initial effort identifies several
teams, which are exploring the following issues: the acquisition/
solicitation process, quality assurance, the liability/claims process,
information systems technology, and electronic billing and payment.
Four of these issues address the recommendations included in DOD‘s
November 2002 report to Congress. The plan identifies a list of
essential activities needed to carry out the responsibilities required
to build the future personal property program. It also includes time
lines and identifies a process to monitor problems and delays. However,
it does not include monitoring costs and benefits during the
implementation phase and the extent the proposed changes are being
achieved within an acceptable and a predefined range. Further, it does
not include evaluating the extent the benefits from the pilot programs
will be achieved after the new program is implemented to determine
whether the anticipated improvements were achieved at a reasonable
cost.
DOD‘s Recommendations Offer Solutions to Long-Standing Problems:
The three recommendations in DOD‘s report to Congress offer solutions
to several of the current program‘s long-standing problems, such as the
liability/claims process and the low quality of service. These problems
have been identified in DOD and GAO‘s prior reports, as well as in
surveys conducted as part of the pilot program evaluation. The
inability to monitor shipments and shipping information has been long
recognized and was highlighted as an additional problem during DOD‘s
evaluation. If implemented, the first recommendation (i.e.,
reengineering the liability/claims process by adopting commercial
practices of minimum valuation, simplifying the filing of claims, and
providing direct settlement with the carrier) has the potential to help
reduce the length of time it currently takes to resolve claims for
lost, destroyed, or damaged household goods because the carrier
recovery time would be eliminated for most moves, increase the
reimbursement rates military personnel receive for their losses, and
reduce DOD‘s claims-related costs. The second recommendation (i.e.,
changing the acquisition process to implement performance-based service
contracts) has the potential to help improve the generally low quality
of service that DOD currently receives from the moving industry. The
third recommendation (i.e., implementing information technology
improvements, which could interface functions across areas such as
personnel, transportation, financial, and claims) has the potential to
improve the program‘s ability to reliably monitor and collect data on
the status and costs of shipments so that accurate reporting can be
provided to DOD and Congress.
Reengineered Liability/Claims Process Has the Potential to Shorten
Time, Increase Reimbursement Rates, and Reduce Claims-Related Costs to
DOD:
As part of its evaluation, the Transportation Command cited that one of
the long-standing problems with military household goods shipments is
the liability/claims process, including the (1) length of time it takes
to resolve claims, (2) low reimbursement rates, and (3) high cost of
claims that DOD must pay.
Length of Time to Resolve Claims:
In a study conducted in 1999, the Military Traffic Management Command
reported that 146 days are expended between the time a claim is filed
by a service member to recovery of costs from the carrier by the
government. During this period, military personnel file their claims
for lost, destroyed, or damaged household goods with their respective
military service‘s claims offices and receive settlements (this occurs,
on average, within 23 days), and then these offices file the claims
against the carriers to recover the costs (this step is completed
within the 146 day period).[Footnote 5] In the Transportation Command‘s
pre-evaluation survey completed in 2000, responses from military
personnel who had recently moved indicated that one of the lowest
performance ratings involved the time required to settle a claim.
Based on the Transportation Command‘s evaluation of the claims process
under the pilot programs, one of the results from implementing the
pilot programs was the 146 day average required under the current
program to settle claims and recover costs was reduced to an average of
30 days since the service member filed directly with the carrier and
the military services did not have to recover costs. Under each pilot
program, military personnel settled claims directly with the carriers.
Service members who were not satisfied with offers made by the carriers
could file their unresolved claims directly with DOD. Military services
worked these claims with the carriers and if a claim was justified, the
service member received just settlement under the pilot programs (i.e.,
if the items were lost or destroyed, the member received full
replacement value, while damaged items were repaired).
Low Reimbursement Rates for Loss and Damage:
In its pre-evaluation survey, the Transportation Command found that
military personnel cited low reimbursement amounts that typically do
not cover the loss or damage of household goods as a major concern
during their moves. Under the current program, a carrier‘s liability is
limited to $1.25 per pound multiplied by the shipment weight. Personnel
receive only the depreciated value of lost, destroyed, or damaged
items, up to a maximum of $40,000 per move.[Footnote 6] When arranging
their moves under the current program, military personnel can buy
increased insurance coverage from their carrier, up to a full
replacement value limit of $3.50 per pound times the shipment weight,
at a cost of 85 cents per $100 of the stated value of the shipment.
However, only military personnel making moves within the continental
United States can buy this additional coverage; it is not available to
those moving to or from overseas posts. Another option that military
personnel have to increase their protection for loss and damage is to
buy additional coverage from private-sector sources.
Unlike the current program, the pilot programs provided full
replacement value for lost and destroyed goods, with maximum amounts
ranging from $63,000 to $75,000. Damaged items were repaired. Two of
the three pilot programs reported that their cost per pound times the
shipment weight rates were $3.50 for $63,000 maximum coverage and $6.00
for $75,000 maximum coverage, respectively. The remaining pilot program
did not give a cost per pound, stating only that its maximum coverage
rate was $72,000.
High Costs to Government for Claims:
DOD has reported that, historically, approximately 35 percent of all
moves result in loss or damage claims. A 1997 Military Traffic
Management Command survey of 3,000 moves revealed that while 65 percent
of shipments had loss or damage, only 35 percent resulted in claims
being filed. DOD pays approximately $100 million a year in claims but
recovers only 60 to 65 percent of the amounts paid to military
personnel from the moving industry. These figures understate the actual
loss and damage, since all military personnel do not file claims,
apparently because the process takes a long time and reimbursement
rates do not always cover the losses. DOD incurs these losses due to
the structure of its current program. The military claims offices
assist service members by arranging to pay their claims and then
submitting the claims to the respective movers for reimbursement. As
indicated above, DOD receives only partial reimbursement from the
moving industry.
If the recommendation is implemented, DOD expects to reduce a
substantial portion of the estimated $100 million it currently pays in
claims each year to service members and eliminate much of the 35 to 40
percent in losses it incurs from settling claims with the moving
industry because service members will be settling claims directly with
their carriers. DOD also expects additional savings because fewer
demands would be placed on military claims officials to manage the
claims process. DOD believes that these savings will help offset the
higher costs of providing full replacement value to service members for
any loss and damage incurred during the shipment and storage of their
personal property.
Change to Performance-Based Service Contracts Could Improve Quality of
Moving Services:
Our work has shown that another long-standing problem with the current
personal property program is the poor quality of moving services
provided to military personnel. The high number of loss and damage
claims that military personnel file underscores this problem. According
to the two pre-evaluation surveys cited in the Transportation Command‘s
evaluation, around 55 to 65 percent of respondents reported suffering
some loss or damage of household goods during a recent move. Moreover,
in the
pre-evaluation survey conducted by the Transportation Command, the top
four factors identified by service members as being of greatest
importance to them in the moving process were the quality of packing,
the care in handling personal property, the condition of their property
upon receipt at the end of the move, and the receipt of fair payment
for any losses or damages they suffered. In individual comments
obtained during the
pre-evaluation survey, the Transportation Command reported that some
service members also cited the lack of professionalism and quality of
customer service on behalf of moving crews as a concern.
The problem stems primarily from the current program placing greater
emphasis on costs (i.e., the lowest bids) than on the quality of
service that carriers provide when moving shipments of military
household goods. While the current program established its Total
Quality Assurance Program to measure quality, data collected to develop
scores for each carrier includes three measures (timeliness of pickup,
timeliness of delivery, and reported loss and damage), which are not
collected for all household goods shipments. The best indication of
quality, customer satisfaction, is not measured in the current program.
The problem of quality is further exacerbated by the program‘s use of a
20 year-old tariff schedule that carriers use in developing their bids.
This tariff contains lower rates than the current commercial tariff
used during the pilot programs.
Unlike the current program, the pilot programs screened carriers that
wanted to participate in their programs by emphasizing the quality of
carriers‘ prior performance rather than the amount of their bids. For
example, the Full Service Moving Project contracted a financial
services company to conduct a financial and performance assessment of
potential movers. The pilot program emphasized best value and placed
more emphasis on performance (70 percent) than cost (30 percent) in
determining which providers were awarded shipments. The pilot programs
showed that these types of contracts could allow the government to
pre-screen carriers for financial viability and, more importantly, to
institute and maintain a quality assurance process to reduce losses and
improve service.
In addition to prescreening carriers for quality control purposes, the
pilot programs also surveyed military personnel who participated in the
programs and used the results to distribute future shipments to
carriers that received the best performance scores. To address concerns
about the obsolete tariff schedule, the pilot programs adopted current
commercial tariffs for carriers to use in establishing their bids.
The pilot programs also showed that the solicitation process could be
streamlined by eliminating detailed statements of work and that the
pilot programs could place responsibility for successful performance on
carriers, allowing the government to focus on outcomes, rather than
processes. Finally, the pilot programs demonstrated that using these
types of performance-based service contracts did not have an adverse
effect on small business participation, a major concern of the moving
industry. On the basis of the total dollar value of shipments, each
pilot program exceeded the Small Business Administration‘s goal of 23
percent participation for the industry. Specifically, 48 percent of the
Military Traffic Management Command‘s Reengineered Personal Property
Program‘s revenues, 74 percent of the Full Service Moving Project‘s
revenues, and 100 percent of the Navy‘s Service Member Arranged Move‘s
revenues went to small businesses.
Implementing Information Technology Improvements to Address Data
Reliability Problems:
Another ongoing problem with the current personal property program is
its inability to provide reliable data on the status of in-transit
shipments or on the number and associated costs of shipments managed by
DOD each year. Because of the lack of reliable data on shipments and
costs, program managers have no way of knowing the actual costs of
moving military personnel‘s household goods. In addition, they have no
access to real-time tracking data that they could use to manage
transportation and storage costs and to help cut down on the need for
temporary storage by reducing the number of failed deliveries.
Two of the pilot programs included features to address the problems
associated with the current program‘s stand-alone data management
system. The pilot programs each developed a Web-based data management
system to enhance the visibility of individual shipments and provide
more reliable data on shipments and costs. For example, the Military
Traffic Management Command‘s Reengineered Personal Property Program‘s
data management system provided in-transit visibility. This made it
possible to track the status of individual shipments and gave real-time
access to those sections of the shipment records that various parties
involved in the relocation process needed for data entry or status
review. The pilot program‘s data management system provided a complete
picture of the service member‘s move from start (the move application
process) to finish (the claims submission and resolution process). In
addition, the data management system demonstrated the potential to
provide information to personnel in various functional areas involved
in the service members‘ relocation process (such as personnel,
transportation, financial, and claims). Finally, the data management
system demonstrated the potential to provide data for planning and
budgeting purposes on the types of shipments made annually across DOD
and their costs. The Full Service Moving Project‘s data management
system was developed but not fully implemented because the military
services terminated their participation in the pilot program due to its
high costs. While the Navy‘s pilot program developed a database near
the end of the Transportation Command‘s evaluation, the database was
not fully implemented nor assessed as part of the evaluation.
Transportation Command‘s Evaluation of Pilot Programs Supports DOD‘s
Three Recommendations:
Our analysis indicated that DOD‘s three recommendations are supported
by the results of the Transportation Command‘s evaluation of the three
pilot programs. The Transportation Command adopted a sound methodology
to conduct its evaluation, and it adjusted this methodology when
circumstances warranted. The results of the Transportation Command‘s
evaluation are based on data collected from a limited number of
geographical areas. While the shipments included in the evaluation do
not represent all shipment types managed annually by DOD, we believe
that the evaluation results provide sufficient information to allow DOD
to initiate actions to improve its current personal property program
within budget constraints.
Transportation Command Implemented a Methodologically Sound
Evaluation:
We found that the Transportation Command used a methodologically sound
approach to evaluate the results of the three pilot programs and make
its recommendations. Before it started the evaluation process, the
Transportation Command considered some lessons learned that had emerged
from our review of the Hunter Pilot Program in 1999,[Footnote 7] and it
followed through with several of them. For example, it obtained
assistance from a contractor to design an evaluation plan that met
professional standards. The Transportation Command identified four
aspects, or factors, of the property program that served as the focus
of its evaluation (i.e., quality of life, total costs, small business
participation, and process improvements). The evaluation plan also
prescribed that only one quality of life survey be administered to each
participating service member in order to avoid survey ’fatigue“ that
can result from subjecting a person to multiple surveys, and thus avoid
the resulting potential for questionable results. In designing the
evaluation plan, the Transportation Command incorporated a number of
important evaluation features. These features included assessing the
four factors consistently across all three pilot programs, ensuring
that the evaluation received data from the pilot programs during the
same time period to avoid the need to make adjustments due to potential
changes in carrier operations and costs, conducting a survey of service
members using the current program to establish a baseline from which to
measure the pilot program results, and developing a method to provide
estimates of what DOD would have paid for comparable shipments under
the current program for those shipments completed under the pilot
programs.
The Transportation Command made appropriate adjustments to the
evaluation plan when it learned that the three pilot programs would not
be underway at the same time and that they would not provide all of the
information originally outlined in the plan. For example, the Full
Service Moving Project began later and terminated earlier than was
expected, the Navy‘s Service Member Arranged Move Pilot Program failed
to conduct a quality of life survey and collect cost data as outlined
in the evaluation plan, and none of the three pilot programs provided
costs associated with individual process improvements. The
Transportation Command included qualitative analytical techniques so
that it could include as much information on each pilot program as
possible in its evaluation while also dealing appropriately with data
limitations. The Transportation Command also shifted the evaluation
focus from the individual pilot programs to specific features from the
three programs, such as full replacement value for loss and damage and
the screening process for carrier participation.
Survey and Analytical Data Support Command‘s Recommendations:
Our work indicated that the Transportation Command‘s analysis of data
collected from the three pilot programs supports the three
recommendations that DOD included in its report to Congress. The
Transportation Command‘s analysis of household goods shipment data from
the pilot programs showed that the average amount of time that service
members and DOD spend to settle claims and recover costs from carriers
fell dramatically in all three pilot programs. In comparison with the
current program‘s 146-day average, it took only 30 days, on average, to
settle a claim under the Reengineered Personal Property Program and the
Full Service Moving Project and fewer than 14 days under the Navy‘s
program. Survey results indicated that full replacement (rather than
depreciated) value, direct claims settlements, and anticipated
improvements in the claims process accounted for the highest increases
in satisfaction. Based on experiences during the pilot programs, DOD
believes that direct claims settlement between service members and
carriers should reduce claims costs DOD currently incurs. Under the
current program, DOD must collect from the carriers after it has paid
the service members‘ claims. DOD expects that this step will be
eliminated in most instances because it is anticipated that service
members will be resolving most of their claims directly with their
carriers.
The Transportation Command‘s analysis of process improvement data,
interviews and observations during site visits, and survey results from
the pilot programs supported DOD‘s recommendation to use performance-
based service contracts to improve the quality of services that the
moving industry provides to the military. The process of prescreening
carriers desiring to participate in the pilot programs on the basis of
their financial viability and past performance helped to eliminate poor
performers. Furthermore, the pilot programs‘ use of post-move surveys
allowed them to get immediate and continuous feedback on the carriers‘
performance and to use this information to distribute future work to
those carriers with the highest performance ratings and best value. In
addition, two of the pilot programs reduced the amount of paperwork
associated with soliciting proposals and approving carriers.
Finally, the Transportation Command‘s review and observations of two of
the pilot programs‘ Web-based data management systems supported DOD‘s
recommendation to overhaul the current personal property program‘s
computer system (the Transportation Operational Personal Property
Standard System). The Transportation Command found that the
Reengineered Personal Property Program‘s data management system
significantly improved communications between the various DOD offices
and the moving industry. The system gave real-time access to shipment
records to DOD‘s personal property shipment offices, certifying
officers, prepayment auditors, military service headquarters, and
military service claims offices and finance centers, as well as moving
industry participants. Similar results occurred with the Full Service
Moving Project‘s Best Value Distribution Database system, but the
military services terminated their participation in this pilot program
before the system‘s full potential could be demonstrated.
While the shipments included in the evaluation do not represent all the
shipment types managed annually by DOD, we believe that the evaluation
results provide sufficient information to allow DOD to initiate actions
to improve its current personal property program.
Ability to Implement New Program within Cost Estimates Reported to
Congress Is Uncertain:
Our review found that the estimates DOD reported to Congress might
understate the total initial cost for implementing the information
technology improvements recommendation and contain a questionable
adjustment for costs associated with the claims and contracting process
recommendations. Also, DOD did not quantify the risk associated with
implementing these latter recommendations within its projected 13
percent increase over the current program‘s cost. Therefore, the
ability to implement changes to the existing program within the cost
estimates reported to Congress is uncertain.
Based on our discussions with Military Traffic Management Command
officials and review of available documents, we concluded that the
total initial cost to implement the information technology improvements
recommendation will more likely be $7 million rather than the $4
million to $6 million estimate that DOD previously reported to
Congress. In its response to a draft of this report, DOD maintained
that the costs to implement a new Web-based data management system
would fall within its initial cost estimate of $4 million to $6
million. DOD‘s projected cost estimate includes $5 million for
development and implementation of the new system and $500,000 each for
user training and system verification and validation testing. At a
minimum, based on these projected cost estimates, the initial cost to
implement the information technology improvements recommendation would
more likely be $6 million.
While we concur with the premise of two of the three adjustments used
to develop the 13 percent cost increase to implement the remaining
recommendations, we are less assured in the extent to which the
projected savings related to the third adjustment may occur. We found
that the first two adjustments were based on historical data. However,
we question the rationale DOD used to develop the third adjustment, as
the savings associated with this adjustment are based on assumed cost
reductions resulting from changes in program operations. Also, these
reductions lack the same quality of evidentiary support as DOD provided
for the other two adjustments.
DOD believes it took a conservative approach in developing the savings
in each of the three adjustments; therefore, it assumes that the
proposed changes to claims and the contracting process can be achieved
within the 13 percent increase over the current program‘s costs. Due to
the long-standing problems with this program and the high pilot program
costs that contributed to the military services‘ early termination of
participating in one of the pilot programs, we believe that by
quantifying the risk associated with this projection, DOD could provide
the military services and Congress information needed to develop and
review future budget requests for this program. Further, without
carefully monitoring costs during the implementation process and
assessing costs and benefits from a period succeeding full
implementation of the recommendations, DOD will not have the
information needed to determine if anticipated improvements in the
program are being achieved at a reasonable cost. Currently, DOD is
beginning planning efforts to implement the recommendations. These
efforts do not include monitoring and evaluating costs and benefits
during the implementation phase and post implementation of the
recommendations in a new program.
Costs to Implement Information Technology Improvements Vary:
The information DOD has provided on costs to implement the information
technology improvements recommendation varies. Information provided
during our review indicated that the total initial cost to improve the
current data management system would be higher than the $4 million to
$6 million DOD included in its report to Congress. DOD worked with the
contractor who developed the Reengineered Personal Property Program‘s
Web-based data management system to develop an estimate of the cost to
expand the capabilities tested during the pilot program. Also included
in this estimate were funds to provide training for users of the new
system. Based on our discussions with officials from the Military
Traffic Management Command and our review of available documents, we
concluded that these costs would more likely be $6 million, as the data
management system development cost was projected to be $5 million with
an additional $1 million for user training. The need for this training
as part of a new personal property program was identified during DOD‘s
evaluation of the pilot programs. We increased our overall projections
for the cost of the new system to $7 million when we learned that DOD
planned to continue spending at least another $1 million annually for
independent verification and validation testing and contractor support.
This latter expense was identified to us during discussions following
DOD‘s submission of its report to Congress.
In its response to a draft of this report, DOD maintained that the
costs to implement a new Web-based data management system would fall
within its initial cost estimate of $4 million to $6 million. It
projected a cost of $5 million for system development and
implementation and $500,000 each for user training and initial system
validation. At a minimum, based on these projected cost estimates, the
initial cost to implement the information technology improvements
recommendation would more likely be $6 million. Because we did not
assess the sufficiency of DOD‘s original estimates of $1 million each
for training and validation testing, we are unable to assess the impact
of the reduction on the improvements in information technology across
DOD.
Based on our discussion with DOD officials, we learned that the plan is
to implement this recommendation regardless of the status of the other
two recommendations because managers and users of the program need more
reliable information to manage the program‘s shipments and their costs.
Funds to implement this recommendation would come from the military
services‘ operations and maintenance accounts.
Soundness of Estimates to Develop Cost for Changing Claims and
Contracting Processes Varies:
The soundness of the three adjustments the Military Traffic Management
Command used to develop its estimated 13 percent increase over the
current program costs to implement the remaining recommendations--the
claims process and performance-based service contracts--varies. We
found that two of these adjustments are based on reasonable assumptions
and are supported by historical experience and by data. The savings
associated with the third adjustment are based on assumed cost
reductions resulting from changes in program operations and lack the
same quality of evidentiary support as DOD provided for the other two
adjustments. Therefore, we are less assured in the extent to which the
savings associated with this adjustment may occur. Finally, we found
that in its report to Congress, DOD did not quantify the risk of
achieving these recommendations within the projected 13 percent
increase. This information is important to the military services as
they develop their military personnel and operations and maintenance
budget requests and to Congress as it assesses the reasonableness of
these requests.
Estimate Includes a Questionable Cost Adjustment:
In developing the 13 percent estimate, the Military Traffic Management
Command determined that three adjustments to the average costs for the
pilot programs were required to develop the cost for the full rollout
of a new personal property program. The first two adjustments (i.e.,
reducing the average weight of shipments and reducing costs to adjust
for a mix of small and large businesses) were made to offset
differences between the pilot programs‘ shipments and those more
typically managed across DOD. The third adjustment was made to reduce
the pilot programs‘ costs to reflect anticipated savings based on
economies of scale.[Footnote 8] In developing these adjustments, the
Military Traffic Management Command worked with a contractor and
consulted with officials from the military services and moving industry
associations.
While we believe that the shipment weight and small business mix
adjustments are reasonable, we question the extent to which the
economies of scale or program efficiencies adjustment may be achieved.
For the weight adjustment, the Military Traffic Management Command
determined that the average weights of moves in the two pilot program
areas were higher than those experienced in typical departmentwide
moves. As a result, the Military Traffic Management Command reduced the
pilot programs‘ average weights to reflect the lower, more typical
weights to be used in calculating a total cost for a departmentwide
program. This adjustment resulted in a 12 percent drop in average
costs. We found the approach of using historical data to more
accurately reflect the typical shipment weights to be reasonable.
Next, the Military Traffic Management Command further lowered the pilot
programs‘ average costs because the pilot programs had higher small
business participation rates than the departmentwide average, and small
businesses are typically more expensive than large businesses. Small
businesses accounted for 48 percent of the cost of all moves under the
Reengineered Personal Property Program and 73 percent under the Full
Service Moving Project. In addition, small businesses were 14 percent
more expensive per shipment in the Reengineered Personal Property
Program and 74 percent more expensive in the Full Service Moving
Project than what each pilot program paid to large businesses. In
developing its departmentwide estimate, the Military Traffic Management
Command used a small business participation target rate of 30 percent.
This 30 percent target rate is higher than the Small Business
Administration‘s 23 percent goal for government agencies conducting
business with this industry. On the basis of this lower participation
rate, the Military Traffic Management Command reduced the pilot
programs‘ average costs further by 8 percent. We agree that this
adjustment in costs based on differences in the pilot programs‘ small
business participation rate and the new 30 percent goal is a reasonable
way to reflect the differences between the pilot programs‘ costs and
the departmentwide-projected costs.
We found that the third adjustment that the Military Traffic Management
Command made--to reduce the cost of departmentwide shipments because of
economies of scale or program efficiencies--was not adequately
supported based on either historical experience or data that DOD later
provided. The Military Traffic Management Command reduced the pilot
programs‘ average costs by 5 percent on the assumption that:
* the pilot programs‘ shipments involved only a limited number of
providers;
:
* the pilot programs only included a limited number of shipments while
the current program manages over 600,000 shipments annually;
:
* more accurate and timely management data that includes service member
counseling, reduced losses, and storage and indirect costs will result
in a more efficient program; and
:
* overhead and operating costs will be spread due to a larger volume of
shipments.
:
While recognizing that some changes may result from these anticipated
program efficiencies, the effect of these changes on potential cost
savings is uncertain at this time. The Military Traffic Management
Command did not provide the same level of evidentiary support that it
provided on the other two adjustments. Further, we believe that only
time will determine if DOD‘s assumption for this adjustment, in
particular, proves to be correct.
DOD Did Not Quantify the Risk for Its Cost Estimate:
We found that DOD has not provided a level of assurance to the military
services and Congress that its projected 13 percent increase over the
current program‘s cost can be achieved. Quantifying the risk associated
with this projection could provide the military services assurance of
the viability of the projected 13 percent increase as they prepare
budgets to support the increased cost for this program. Congress could
also use this information as it reviews DOD‘s requests for additional
funds to implement changes in this program. The need for this type of
information is further supported based on the long-standing problems
associated with the current program and the fact that shipment and
storage costs under the pilot programs were significantly higher than
those that DOD estimated it would have paid under its current program
in the same geographical areas. These costs ranged from 31 to 32
percent higher under the Reengineered Personal Property Program and
from 51 to 54 percent higher under the Full Service Moving
Project.[Footnote 9] These higher-than-anticipated costs contributed
to the military services‘ decision to terminate their participation in
the Full Service Moving Project before its test period ended.
While DOD did not quantify the risk, per se, it believes a conservative
approach was taken in developing the savings in each of the three
adjustments. As a result, DOD assumes that the proposed changes to the
claims and contracting processes can be achieved with its projected
increase of 13 percent over the current program‘s budget. We still
believe that the Military Traffic Management Command could have
quantified the risk and provided this additional information to the
military services and Congress as additional assurance of the
likelihood of implementing the two recommendations within its projected
13 percent increase. The need for this information is further supported
based on the long-standing problems DOD has experienced in this
program, the fact that the military services terminated participation
in one of the pilot programs due to the high cost increases, and the
need to determine whether the proposed additional funds from military
personnel and operations and maintenance accounts will be sufficient to
implement the recommendations.
In addition to the information that could be gained from quantifying
the risk of its cost projection, we believe that only by careful
monitoring during the implementation phase will DOD be able to ensure
that the proposed changes are being achieved within an acceptable and a
predefined range. Further, while we believe that the evaluation results
support implementing plans to enhance the current program, it should be
noted that the pilot programs‘ shipments included in the evaluation
were not typical of all types of shipments managed annually. Therefore,
DOD was precluded from projecting the extent to which the recommended
improvements can be achieved DOD-wide. Unless a subsequent evaluation
is undertaken after the recommendations have been implemented, DOD will
not be able to assess the extent to which the projected benefits are
being achieved for military personnel, their families, and DOD, and
whether the benefits are being achieved at a reasonable cost. Selecting
an evaluation period to include the peak-moving season would also
provide DOD with the information its needs to determine if the proposed
changes can be achieved during the summer, when the demand for moving
services by DOD and the private sector is at its highest.
Conclusions:
The three recommendations DOD developed from its evaluation of the
current and pilot programs, if implemented successfully, could enhance
the quality of life for relocating service members and their families;
reduce claims-related costs to DOD; and resolve problems related to the
reliability of management information on the status of shipments and on
the quantity, types, and costs of shipments that DOD and the military
services manage annually. Delaying implementation of the
recommendations only prolongs problems military personnel, their
families, and DOD experience under the current program.
DOD has not quantified the risk associated with achieving its projected
13 percent increase over the current program‘s costs to implement the
claims process and performance-based service contract recommendations.
Without quantifying the risk, the military services and Congress cannot
be assured that these recommendations can be achieved within this
estimate or whether additional funding or trade-offs may be needed.
Further, without careful monitoring during the implementation phase,
DOD will not be able to ensure that the proposed changes are being
achieved within an acceptable and a predefined range.
Because the pilot programs‘ shipments included in the evaluation were
not typical of all types of shipments managed annually, it was not
possible for DOD to project the extent to which the recommended
improvements can be achieved departmentwide. Without evaluating the
program following implementation of the recommendations, DOD will be
unable to assess the extent to which the projected benefits for
military personnel, their families, and DOD are being achieved and, if
so, whether they are being achieved within a reasonable cost. Also, if
DOD does not select an evaluation period that includes the peak-moving
season, it will not have the information needed to determine if the
proposed changes can be achieved in the summer, when the demand for
moving services is at its highest.
Recommendations for Executive Action:
To improve the personal property program for military personnel, their
families, and program administrators, we recommend that the Secretary
of Defense direct the Commander, U.S. Transportation Command, to:
* initiate actions to implement the three recommendations contained in
DOD‘s report to Congress within budget constraints,
:
* provide the military services and Congress additional information to
quantify the risk associated with achieving the projected 13 percent
cost estimate before the claims process and performance-based service
contracts recommendations are implemented to provide the military
services with information needed for budgeting purposes,
:
* monitor costs for all recommendations during the implementation phase
to ensure that the proposed changes are being achieved within an
acceptable and a predefined range, and
:
* assess the effects of the three recommendations on the personal
property program (to be carried out after the summertime peak-moving
season once the recommendations have been implemented) to determine
whether the anticipated improvements in the program are being achieved
at a reasonable cost.
Agency Comments and Our Evaluation:
In commenting on a draft of this report, DOD concurred with three of
our four recommendations. For the first of these recommendations, DOD
stated that it is developing a plan to implement those recommendations
it reported to Congress and anticipates its recommendations will be
implemented by the end of the first quarter of fiscal year 2006,
assuming the military services receive the additional funds needed to
fund program enhancements. In response to our recommendation to monitor
costs during the implementation phase, DOD stated that rolling out the
new program will require monitoring of costs to determine if the moving
industry partners are submitting bids that will allow DOD to enhance
this program within the projected 13 percent cost increase. Further,
DOD plans to include a process to conduct a rate reasonableness
analysis upon receipt of the rates. For rates found to be outside the
range of reasonableness, carriers will be given one opportunity to
resubmit their rates. DOD plans to only use those rates determined to
be reasonable in the new program. DOD also plans to include metrics,
target/benchmark performance indicators, and a methodology for data
collection in an updated program of action and milestone plan. For our
recommendation, i.e., assess the effects of the three DOD
recommendations on the personal property program to determine whether
the anticipated improvements in the program are being achieved at a
reasonable cost, DOD plans to collect data needed to determine if
anticipated improvements have been achieved on a continuing basis. DOD
plans to use customer satisfaction surveys in developing carrier
performance ratings, which will be established quarterly, with the
exception of the peak season, when performance ratings will be
established monthly. If properly implemented, we believe the proposed
DOD actions will sufficiently address these recommendations.
DOD partially concurred with the remaining recommendation, i.e.,
provide the military services and Congress with additional information
to quantify the risk associated with achieving the projected 13 percent
cost estimate to provide the military services with information needed
for budgeting purposes. DOD continues to believe that the 5 percent
reduction it made to pilot programs‘ average costs to adjust for
economies of scale/program efficiencies was reasonable and very
conservative and that the program can be implemented within the
projected 13 percent increase over current program costs. DOD also
reported that one of the military services validated the 13 percent
cost increase following our audit. Further, DOD stated that it did not
see value added in providing the military services or Congress a formal
risk assessment but will continue to work with the military services as
execution progresses to make sure they have all information required
for budget purposes. Additionally, while not part of this
recommendation, DOD also said it did not concur with our finding that
the cost estimate for implementing its information technology
improvements recommendation would be $7 million.
In reviewing the response, we found that DOD still did not provide any
data to support its assumption of a 5 percent cost savings from
economies of scale/program efficiencies. DOD stated that the new
program will be about 200 times larger than the pilot programs and that
the resulting increase in volume will lower the cost per unit, a
standard and accepted law of economics. While we agree that the cost
may decrease, it may also increase or remain unchanged. Moreover, the
cost may decrease by less than 5 percent. Without specific data showing
the per move costs will decrease as the scale of operations increase,
we continue to question the basis for DOD‘s assumption of a 5 percent
reduction.
We believe that the validation effort completed by one of the military
services, along with the calculations and assumptions DOD used in
developing the 13 percent cost estimate, does not provide the military
services and Congress with information needed to reliably develop and
review budget requests to fund enhancements to the current program. We
continue to believe that DOD needs to qualify this estimate with a
measure of the risk associated with implementing its recommendations.
Without providing the range of possible cost increases and the risk
regarding the likelihood of achieving this 13 percent projection within
that range, DOD may encounter a repetition of its experience with one
of the pilot programs, which had to be terminated because actual costs
exceeded projected costs. Absent this risk information, the military
services will have to wait until after the transportation providers
submit their bids in order to learn whether the recommendations can be
implemented within the 13 percent projection. Should the bids result in
costs that exceed this estimate, DOD and the military services will
need to make adjustments to ensure that the recommendations are
implemented within funding limits. Therefore, we continue to believe
that our recommendation has merit.
Our finding that the implementation of the information technology
improvements recommendation would likely cost $7 million rather than
the $4 million to $6 million that DOD projected was based on
information we received from DOD during the audit. Specifically, we
calculated that the costs to develop and implement the new system would
be about $5 million and that training for users of the enhanced system
would cost an additional $1 million. DOD had identified the need for
this training during its evaluation of the pilot programs. After DOD
submitted its report to Congress, it identified another potential cost-
-an additional $1 million for independent verification and validation
testing of the system. Our $7 million estimated included all three of
these cost elements. In its response to a draft of this report, DOD
maintained that its costs estimate would fall within its initial cost
estimate of $4 million to $6 million, including $5 million for system
development and implementation and an additional $500,000 each for user
training and system verification and validation testing. At a minimum,
based on these projected cost estimates, the initial cost to implement
the information technology improvements recommendation would more
likely be $6 million. However, since we did not originally assess the
sufficiency of the $1 million estimates for training and testing, we
are unable to assess what impact DOD‘s reduction for these costs to
$500,000 would have on the implementation of the system across DOD. We
have reflected DOD‘s changes in the body of our report.
DOD‘s comments are reprinted in appendix III. DOD also provided
technical comments, and we revised our report to reflect them where
appropriate.
We performed our review from April 2002 through February 2003 in
accordance with generally accepted government auditing standards.
Appendix I contains the scope and methodology for this report. DOD‘s
comments are reprinted in their entirety in appendix III.
We are sending copies of this report to the appropriate congressional
committees; the Secretary of Defense; the Commander, U.S.
Transportation Command; and the Director, Office of Management and
Budget. We will also make copies available to others upon request. In
addition, the report will be made available at no charge on the GAO Web
site at http://www.gao.gov.
Please contact me at (202) 512-8365 or Lawson Gist, Jr., at (202) 512-
4478 if you or your staff have any questions concerning this report.
Key contributors to this assignment were Robert L. Self, Jacqueline S.
McColl, Arthur L. James, Jr., Charles W. Perdue, and Nancy L. Benco.
William M. Solis
Director
Defense Capabilities and Management:
Signed by William M. Solis:
List of Congressional Committees:
The Honorable John Warner
Chairman
The Honorable Carl Levin
Ranking Minority Member
Committee on Armed Services
United States Senate:
The Honorable Ted Stevens
Chairman
The Honorable Daniel K. Inouye
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
United States Senate:
The Honorable Duncan Hunter
Chairman
The Honorable Ike Skelton
Ranking Minority Member
Committee on Armed Services
House of Representatives:
The Honorable Joel Hefley
Chairman
The Honorable Solomon P. Ortiz
Ranking Minority Member
Subcommittee on Readiness
Committee on Armed Services
House of Representatives:
The Honorable Jerry Lewis
Chairman
The Honorable John P. Murtha
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
House of Representatives:
[End of section]
Appendix I: Scope and Methodology:
To assess the extent to which the recommendations in the Department of
Defense‘s (DOD) November 2002 report to Congress addressed major
problems in the personal property program, we took the following steps:
* To identify the major problems facing the current personal property
program, we reviewed DOD and GAO reports addressing this program. These
reports identified problems associated with quality of service and
claims. We also conducted interviews with personal property program
officials and their contractors to gain an understanding of the current
data management system‘s limitations and the long-standing problems
involving the lack of reliable information on shipments and their
costs.
:
* To determine whether the proposed recommendations in DOD‘s report to
Congress addressed the major problems of the current program, we
tracked the recommendations back to the U.S. Transportation Command‘s
report on its evaluation results and assessed the extent to which the
recommendations are linked to and have the potential to address
problems.
:
To assess whether the recommendations in DOD‘s report to Congress were
supported by DOD‘s evaluation findings and should be implemented, we
took the following steps:
* To determine if the Transportation Command developed a
methodologically sound evaluation plan, we assessed the command‘s
efforts against the findings and recommendations contained in our
report[Footnote 10] on the Army‘s Hunter Pilot Program results and
against professional standards[Footnote 11] we would use if conducting
a comparable evaluation. These sources addressed issues such as (1)
seeking advice in designing a methodologically sound evaluation plan,
(2) developing the evaluation plan prior to testing, (3) identifying
factors to be assessed and the data required for analyses to develop
findings and recommendations, (4) limiting quality of life surveys to
only one for each participant to preclude survey ’fatigue,“ and (5)
conducting simultaneous testing of the pilot and current programs.
* To determine if the Transportation Command implemented an effective
evaluation strategy during the data collection phase of its evaluation,
we reviewed the pilot programs‘ efforts to collect data for the four
factors as prescribed in the Transportation Command‘s evaluation plan.
We also assessed the adjustments the Transportation Command made in its
evaluation strategy to address issues that could affect the soundness
of the results. An example of the issues addressed included developing
a constructed cost methodology to provide better estimates of what DOD
would have paid under the current program for shipments made by the
pilot programs.
:
* To assess the Transportation Command‘s development of findings and
recommendations to improve the current personal property program, we
reviewed the evaluation techniques (quantitative and qualitative
analyses) used to analyze data collected for the four factors. Further,
we assessed the extent to which the Transportation Command adjusted the
evaluation techniques to make up for differences in the way that the
pilot programs provided data for the evaluation.
:
To assess the methodology that DOD used to develop cost estimates for
implementing the recommendations, we took the following steps:
* To determine the reliability of the cost estimates for the pilot
programs and for the proposed recommendations, we reviewed the cost
projection methodologies used by the Transportation Command and by the
Military Traffic Management Command.
:
* To determine the reliability of pilot program shipment-related costs
used in the report, we reviewed the data collection efforts used by
each pilot program for the transportation and storage of household
goods included in the Transportation Command‘s evaluation. Further, we
reviewed the constructed cost methodology used to develop the estimates
of what DOD would have paid to make comparable shipments under the
current program in the pilot programs‘ test areas.
:
* To determine the reasonableness of the assumptions and sources of
data used to develop cost estimates for implementing recommendations
for the personal property program, we met with officials from the
Military Traffic Management Command and their contractor to discuss the
methodology. We also reviewed the contents of their briefing on the
cost estimate work for implementing changes to the claims process and
performance-based service contracts and
additional information the Military Traffic Management Command provided
on the costs to implement information technology improvements.
:
We did not make an assessment of whether the anticipated benefits to be
derived from implementing the three recommendations would warrant the
additional costs DOD projects will be required to fund these
improvements. Furthermore, we did not independently test the
reliability of data DOD extracted from its data system to develop
costs. We found that the department placed proper caveats on their use
of such data, and in the case of comparing pilot programs‘ shipment
costs to current program costs, developed a constructed cost
methodology to address current program data management system
weaknesses.
During this and prior[Footnote 12] reviews of DOD‘s evaluation efforts,
we met with officials and obtained documents from the Office of the
Assistant Deputy Under Secretary of Defense (Transportation Policy),
Washington, D.C.; the U.S. Transportation Command, Scott Air Force
Base, Illinois; the Military Traffic Management Command, Alexandria,
Virginia; the Department of Defense Inspector General, Full Service
Moving Project, and Hay Group (Transportation Command Contractor),
Arlington, Virginia; American Management Systems (Transportation
Command contractor), PricewaterhouseCoopers (Military Traffic
Management Command contractor), and Systems Research and Applications
(Military Traffic Management Command contractor), Fairfax, Virginia;
Logistics Management Institute (Military Traffic Management Command
contractor), McLean, Virginia; the Navy‘s Service Member Arranged Move
Pilot Program, Mechanicsburg, Pennsylvania; The Gallup Organization
(Full Service Moving Project contractor), Omaha and Lincoln, Nebraska;
and Parsifal Corporation (Military Management Traffic Command
contractor), Palm Bay, Florida. In addition to these agency meetings
and documents, we drew upon information contained in a testimony
statement, in reports, and in status briefings resulting from our prior
reviews of this program.
Our work for this review was performed from April 2002 through February
2003 in accordance with generally accepted government auditing
standards.
:
[End of section]
Appendix II: Overview of Current Personal Property Program and Pilot
Programs:
The Transportation Command evaluated three pilot programs to assess
alternative approaches that might address long-standing problems with
its current personal property program. The following tables provide
features of the current program and the three pilot programs. As the
tables show, the pilot programs had several features that provided
enhancements to military personnel and their families and to DOD that
are not offered by the current program.
Table 1 compares claims-related features. Specifically, the pilot
programs provided full replacement value rather than depreciated value
for loss and damage and guaranteed claims settlement with 45 to 60 days
of filing the claims.
Table 1: Claims-Related Features of the Current Personal Property
Program and Pilot Programs:
Program features: Loss and damage claims.
Program features: Reimbursements: Current personal property program:
[Empty]; Military Traffic Management Command‘s Reengineered Personal
Property Program: [Empty]; Department of Defense‘s Full Service Moving
Project: [Empty]; Navy‘s Service Member Arranged Move Pilot Program:
[Empty].
Program features: --Basis for valuing property; Current personal
property program: Depreciated value; Military Traffic Management
Command‘s Reengineered Personal Property Program: Full replacement
value; Department of Defense‘s Full Service Moving Project: Full
replacement value; Navy‘s Service Member Arranged Move Pilot Program:
Full replacement value.
Program features: --Maximum dollar value per move; Current personal
property program: $40,000; Military Traffic Management Command‘s
Reengineered Personal Property Program: $63,000; Department of
Defense‘s Full Service Moving Project: $75,000; Navy‘s Service Member
Arranged Move Pilot Program: $72,000.
Program features: Guaranteed claims settlement; Current personal
property program: Not specified; Military Traffic Management Command‘s
Reengineered Personal Property Program: Within 60 days; Department of
Defense‘s Full Service Moving Project: Within 45 days; Navy‘s Service
Member Arranged Move Pilot Program: Within 60 days.
Sources: DOD (data); GAO (analysis).
[End of table]
Table 2 compares the quality of service-related features. Some of the
comparable features included emphasizing performance over cost in
selecting transportation providers and prescreening of transportation
providers.
Table 2: Quality of Service-Related Features of the Current Personal
Property Program and Pilot Programs:
Program features: Counseling and arranging shipment services.
Program features: Single relocation coordinator; Current personal
property program: Counseling and arranging shipment services: No;
Military Traffic Management Command‘s Reengineered Personal Property
Program: Counseling and arranging shipment services: No; Department of
Defense‘s Full Service Moving Project: Counseling and arranging
shipment services: Yes; Navy‘s Service Member Arranged Move Pilot
Program: Counseling and arranging shipment services: Yes.
Program features: Program management and counseling services provider;
Current personal property program: Counseling and arranging shipment
services: Installation personal property shipping office; Military
Traffic Management Command‘s Reengineered Personal Property Program:
Counseling and arranging shipment services: Installation personal
property shipping office; Department of Defense‘s Full Service Moving
Project: Counseling and arranging shipment services: Private-sector
move managers; Navy‘s Service Member Arranged Move Pilot Program:
Counseling and arranging shipment services: Installation personal
property shipping office.
Program features: Screening process for transportation providers.
Program features: Program emphasis in selecting transportation
providers; Current personal property program: Counseling and arranging
shipment services: Lowest cost; Military Traffic Management Command‘s
Reengineered Personal Property Program: Counseling and arranging
shipment services: Performance; Department of Defense‘s Full Service
Moving Project: Counseling and arranging shipment services:
Performance; Navy‘s Service Member Arranged Move Pilot Program:
Counseling and arranging shipment services: Performance.
Program features: Prescreening of transportation providers; Current
personal property program: Counseling and arranging shipment services:
Very limited; Military Traffic Management Command‘s Reengineered
Personal Property Program: Counseling and arranging shipment services:
Yes; Department of Defense‘s Full Service Moving Project: Counseling
and arranging shipment services: Yes; Navy‘s Service Member Arranged
Move Pilot Program: Counseling and arranging shipment services: Yes.
Program features: Customer satisfaction surveys conducted and method;
Current personal property program: Counseling and arranging shipment
services: Surveys are not conducted; Military Traffic Management
Command‘s Reengineered Personal Property Program: Counseling and
arranging shipment services: Survey company calls member; Department of
Defense‘s Full Service Moving Project: Counseling and arranging
shipment services: Survey company calls member; Navy‘s Service Member
Arranged Move Pilot Program: Counseling and arranging shipment
services: Service member returns survey via mail.
Program features: Visibility of shipments during relocation process.
Program features: Methods used to increase service members‘ visibility
of shipments during move process; Current personal property program:
Counseling and arranging shipment services: None; Military Traffic
Management Command‘s Reengineered Personal Property Program:
Counseling and arranging shipment services: Toll-free number to
transportation provider; Department of Defense‘s Full Service Moving
Project: Counseling and arranging shipment services: Toll-free number
to move manager and transportation provider; Navy‘s Service Member
Arranged Move Pilot Program: Counseling and arranging shipment
services: Toll-free number to move coordinator and transportation
provider, and member has pager.
Sources: DOD (data); GAO (analysis).
[End of table]
Table 3 compares data reliability-related features. As noted, only one
of the pilot programs had a data management system that provided
reliable information to track individual shipments in transit and
provide overall data on shipments and their associated costs.
Table 3: Data Reliability-Related Features of the Current Personal
Property Program and Pilot Programs:
Program features: Availability and reliability of data on household
goods shipments.
Program features: Reliable data management system to track individual
shipments in transit and to provide overall data on shipments and
associated costs; Current personal property program: Availability and
reliability of data on household goods shipments: No (current system is
not designed to track shipments nor provide reliable shipment and cost
data); Military Traffic Management Command‘s Reengineered Personal
Property Program: Availability and reliability of data on household
goods shipments: Yes; Department of Defense‘s Full Service Moving
Project: Availability and reliability of data on household goods
shipments: Not determined (developed but not fully operational-data
management system needed additional refinement); Navy‘s Service Member
Arranged Move Pilot Program: Availability and reliability of data on
household goods shipments: Not determined (developed a database by end
of pilot program but was not fully implemented or evaluated).
Sources: DOD (data); GAO (analysis).
[End of table]
Additional information on the current program and on each pilot program
and its unique features follows.
Current Program:
The current DOD personal property program, valued at over $1.7 billion
annually, moves more than 600,000 shipments each year for military
personnel and their families from the military services, Defense
agencies, and the Coast Guard. DOD is the moving industry‘s single
largest customer. Managed centrally by the headquarters office of the
Military Traffic Management Command and administered locally by about
200 military and DOD transportation offices around the world, this
program relies on over 1,200 domestic commercial carriers and more than
150 forwarders for international traffic to provide moving and storage
services.
Loss and Damage Claims:
When loss and damage occur, military personnel can submit claims to
their respective military service claims office. Based on depreciated
values, the reimbursement rate is $1.25 per pound multiplied by the
shipment weight, with a maximum amount of $40,000 per move. Military
personnel have up to 2 years after receiving their shipments to file
claims but must submit notice of loss and damage within 70 days of
delivery. The current program does not have a specified time period in
which the claims are to be settled.
Counseling and Arranging Shipment Services:
The current program provides counseling services and arranges the
shipment and storage of household goods and unaccompanied baggage
through government representatives, who are available to assist
military personnel and their families at the origin and destination
points of their moves. The current program does not have a real-time
tracking system for shipments nor does it provide a single point of
contact to manage the entire moving process; therefore, military
personnel may interact with several people at the origin and
destination offices during their relocation.
Screening and Shipment Distribution Process for Transportation
Providers:
The current system is not designed to select transportation providers
on the basis of quality service; rather, transportation providers
offering a minimally acceptable level of quality are generally selected
based on the lowest rates. The program uses the Total Quality Assurance
Program to develop quality scores for each transportation provider.
Each local military installation distributes its traffic using a
traffic distribution roster. Transportation providers are placed on the
rosters for each channel (origin and destination areas) by order of
rate level and quality score. Transportation providers who participate
in the domestic part of the current program submit their rates as a
percentage of the government tariff, which is nearly 20 years old. The
providers who participate in the international part of the current
program submit single factor or fixed rates per hundredweight of the
shipments.
The current program does not use customer satisfaction surveys as a
means to evaluate transportation provider performance. To remain in the
program, a provider must maintain a minimally acceptable level of
quality-a 90 percent score. Three factors are measured: on-time pickup,
on-time delivery, and reported loss and damage to determine if points
should be deducted from transportation providers and allocation of
shipments should be reduced or terminated.
Visibility of Shipments during the Relocation Process:
The current program does not provide service members with real-time
visibility of shipments during the relocation process.
:
Availability and Reliability of Data on Household Goods Shipments:
Personnel at origin and destination personal property shipping offices
enter information on shipments to their respective Transportation
Operational Personal Property Standard Systems. However, data in these
individual systems does not include all shipments that occur during the
year, and the systems are not accessible to all parties involved in the
relocation process. Destination personal property shipment offices are
forwarded information on shipments via the current system; however,
payment data on these shipments is maintained in a separate system.
In addition to not providing information on all aspects of individual
shipments, the current program‘s data management system does not
provide DOD and the military services information about the types of
shipments and related costs for planning and budgeting purposes. The
following are examples of the current system‘s limitations:
* the format of the system makes compiling data from multiple sites
difficult;
* not all data is captured promptly; and:
* not all data and costs are captured/updated in the system.
Funding Sources:
Under the current program, the military services reimburse carriers and
forwarders for shipment-related costs from military personnel accounts.
Personal property shipment office expenses and claims filed with the
government are funded from the military services‘ operations and
maintenance accounts.
Military Traffic Management Command‘s Reengineered Personal Property
Program:
Sponsored by the Military Traffic Management Command, the Reengineered
Personal Property Program included outbound shipments for military and
Coast Guard personnel departing from installations located in North
Carolina, South Carolina, and Florida (excluding Tyndall Air Force
Base). The pilot program ran concurrently with the existing program at
these installations. The pilot program‘s goal was to include 50 percent
of eligible moves from the above installations to continental United
States and European locations. The remaining shipments were to be moved
under the existing program. The Reengineered Personal Property Program
was initiated in January 1999 and operated for 12 months before data
was submitted to the Transportation Command for evaluation.
Loss and Damage Claims:
Reimbursement for loss and damage claims was increased from depreciated
value to full replacement value, and the dollar amounts per move
increased from $40,000 under the current program to $63,000 under the
Reengineered Personal Property Program. Additionally, the pilot program
provided direct claims settlement between military personnel and their
transportation providers and a requirement that transportation
providers settle claims within 60 days of receiving claims forms from
military personnel.
Counseling and Arranging Shipment Services:
Like the current program, the Reengineered Personal Property Program
relied on personnel in the personal property shipping offices to
provide counseling services and arrange for shipment and storage of
household goods and unaccompanied baggage. A central contact point in
these offices was not designated to manage the entire moving process;
therefore, military personnel may have interacted with several people
at the origin and destination offices during the relocation process.
However, to improve customer service, the program‘s Pilot
Transportation Operational Personal Property Standard System provided
real-time worldwide tracing capability.
Screening and Shipment Distribution Process for Transportation
Providers:
Greater emphasis was placed on performance in awarding shipments to
transportation providers. Evaluation of financial status, elimination
of high-risk companies, and consideration of providers‘ past
performance, rather than lowest bid, played the dominant role in
selecting initial transportation providers to participate in this pilot
program. Transportation providers who participated in the pilot program
submitted their bids for various origin and destination routes as a
discount from the commercial tariff. Prices were fixed for a year, with
no provision for rate increases during the contract period. Awards were
made only to transportation providers whose offers conformed to the
solicitation and represented the best overall value to the government.
The Military Traffic Management Command evaluated company performance
quarterly and compliance with terms and conditions of the contracts
annually. Subsequent performance reviews were conducted based on
customer satisfaction surveys and claims data. After transportation
providers received their minimum guarantee of business for the year
($25,000), future awards were offered to the best performers. Feedback
was provided monthly to transportation providers, and those that became
poor performers were no longer offered household goods and
unaccompanied baggage shipments.
Visibility of Shipments during the Relocation Process:
The Reengineered Personal Property Program provided the transportation
provider‘s toll-free number to military personnel to enhance visibility
over their shipments throughout the relocation process.
Availability and Reliability of Data on Household Goods Shipments:
The Reengineered Personal Property Program implemented its central,
Web-based Pilot Transportation Operational Personal Property Standard
System in part to address problems associated with visibility of and
availability of information on shipments during the relocation process.
The pilot program‘s data management system provided real-time access to
both shipment and payment records. Access to the various modules of the
system was granted to personal property shipment office personal at
origin and destination locations, transportation providers, invoice
certifying officers, prepayment auditors, military service
headquarters, and military service claims offices and finance centers,
based on each party‘s need for the information.
The system‘s design allowed for entry of current address and telephone
numbers of military personnel to improve the process of delivering
household goods to a new residence. Data reliability was enhanced under
the Reengineered Personal Property Program, but one problem noted
during the evaluation was the need for military personnel to ensure
that their contact information (phone number and address) was current
during the relocation process. This had an effect on deliveries of
household goods and the quality of life survey contractor‘s ability to
reach military personnel to ascertain their opinions about their
relocation experience.
In addition to providing information on all aspects of individual
shipments, the Reengineered Personal Property Program‘s data management
system demonstrated the potential to provide DOD and the military
services with information about the types of shipments and related
costs managed annually for planning and budgeting purposes.
Other Reengineered Personal Property Program Features:
The Reengineered Personal Property Program achieved stronger
transportation provider commitment with long-term contracts, and it
used contractor support to conduct quality of life surveys with
military personnel moving under the pilot program and to perform audits
of each invoice submitted by transportation providers.
Funding Sources:
Like the current program, the military services reimbursed carriers and
forwarders for shipment-related costs from their military personnel
accounts. Transportation office expenses and any claims filed with the
government were funded from the services‘ operations and maintenance
accounts.
The Department of Defense‘s Full Service Moving Project:
Sponsored by the Office of the Assistant Deputy Under Secretary of
Defense (Transportation Policy), the Full Service Moving Project
included outbound shipments for military and Coast Guard personnel and
DOD civilian departing from locations in the National Capital Region,
Georgia (excluding Robins Air Force Base), and Minot Air Force Base,
North Dakota. The pilot program‘s goal was to include 90 percent of the
moves from these locations to continental United States and to European
and Asian-Pacific locations. The remaining shipments were to be moved
under the current program. The Full Service Moving Project began in
January 2001 and continued until its early termination in September
2001. Due to continuing delays in implementing this pilot program and
DOD‘s decision to terminate the pilot program in September 2001, the
Full Service Moving Project had limited operational experience before
submitting data to the Transportation Command.
Loss and Damage Claims:
Reimbursement for loss and damage claims was increased from depreciated
value to full replacement value, with the dollar amounts increasing
from $40,000 per move under the current program to $75,000 per move
under the Full Service Moving Project. Additionally, the pilot program
provided for direct claims settlement between military personnel and
their transportation providers and a requirement that the responsible
party (transportation providers or move managers) settle claims within
45 days of receiving claim forms from military personnel.
Counseling and Arranging Shipment Services:
Unlike the current program and other pilot programs, the Full Service
Moving Project tested the use of private-sector relocation companies
(move managers) for outsourcing traditional transportation services
(counseling and arranging for the shipment and storage of household
goods and unaccompanied baggage) performed by origin and destination
personal property shipping offices. The pilot program‘s goal was to
provide a single point of contact (move manager) for military personnel
and transportation providers to contact throughout the relocation
process.
Screening and Shipment Distribution Process for Transportation
Providers:
The Full Service Moving Project made major changes to the existing
transportation provider approval, rate solicitation, and traffic
distribution processes. The pilot program emphasized best value and
placed more weight on performance (70 percent) than cost (30 percent)
in determining which transportation providers would be awarded
shipments. The pilot program contracted with a financial services
company to conduct financial and performance assessments of
transportation providers and move manager companies that wanted to
participate in the pilot program. For approved transportation
providers, rates were established for a 1-year cycle. The providers
submitted their rates as a discount from the commercial tariff for
domestic shipments and negotiated single rate factors for the overseas
locations. Approved move managers were awarded 2-year contracts with 1-
year options. The move management companies competitively bid their
fees as flat rates, depending on whether they were responsible for
claims settlement or the transportation provider carried this
liability. Also, different fees were established for domestic and
international shipments.
The Full Service Moving Project used survey data from all personnel
participating in the pilot program to determine future percentages of
shipments that would be allocated to the transportation providers. The
pilot program also planned to use survey data on move manager
performance to determine future participation in the pilot program and
incentive payments. The Full Service Moving Project‘s Web-based Best
Value Distribution Database maintained the transportation providers‘
quality and cost scores based on survey information and costs
associated with prior shipments. Move managers used this data to assign
future shipments. However, in some instances (i.e., for group moves,
when meeting small business requirements, when there was a lack of
transportation provider capacity to handle shipments offered, for
multiple shipments to a single transportation provider, and for
international shipments to areas without an established rate), move
managers were told to deviate from the information provided by the data
management system.
Visibility of Shipments during the Relocation Process:
One of the goals of incorporating move managers into the relocation
process was to provide real-time information to military personnel and
to transportation providers regarding the status of household goods
shipments. The move managers, unlike the current program‘s personal
property shipping office personnel, were responsible for the entire
relocation process from the point of origin in establishing
entitlements, arranging for transportation providers, and handling
other personnel-related issues, to the destination in overseeing
deliveries, approving storage, and either settling claims or assisting
military personnel with issues involving settling claims with the
transportation providers if the liability fell with the providers.
Working with both military personnel and transportation providers, the
move managers used contact information to keep military personnel
informed of their shipments‘ status and to coordinate the delivery of
the shipments at the destination. Additionally, as part of the pilot
program, all participants were provided a toll-free number to maintain
visibility over their shipments throughout the process.
Availability and Reliability of Data on Household Goods Shipments:
In addition to move managers, the Full Service Moving Project‘s Web-
based Best Value Distribution Database was implemented to address
problems associated with visibility of shipments during the relocation
process. The pilot program‘s data management system had access to both
shipment and payment records via interface with US Bank‘s PowerTrack
and the move managers‘ systems. Access to the pilot program‘s data
management system was granted to move managers, invoice certifying
officers, military service headquarters, and military service claims
offices and finance centers, based on each party‘s requirements.
Move managers were responsible for keeping the status of the shipments
current in the pilot program‘s data management system. However, the
move managers did not always update this information in the system.
Further, the ability of the move manager to contact the service member
was directly affected by the information provided by the member.
The Full Service Moving Project‘s Web-based Best Value Distribution
Database was anticipated to provide DOD and the military services
information on the types of shipments and related costs managed
annually for planning and budgeting purposes. Unlike the Reengineered
Personal Property Program where various parties in the relocation
process entered data into that pilot program‘s data management system,
the majority of the data in the Full Service Moving Project‘s data
management system was predicated on the move managers gathering and
entering the information.
Other Full Service Moving Project Features:
The Full Service Moving Project achieved stronger transportation
provider commitment with long-term contracts and faster payment of
invoices; it offered binding cost estimates for shipments; it used
contractor support to conduct quality of life surveys with military
personnel moving under the pilot program and to perform audits of each
invoice submitted by the transportation providers; and it offered
optional relocation referral assistance for activities such as the
purchase and sale of service members‘ residences.
Move managers were required to perform prepayment audits and business
rules were established for an automatic payment method. Payment
methodology was predicated on the move manager entering the expected
invoice into PowerTrack and the transportation provider submitting a
notice of delivery and invoice. Payment timeliness was also driven by
the timeliness of documentation submitted by the transportation
providers. On some invoices, the contracting representative had to
review and certify payment in PowerTrack. This occurred when the match
showed a difference of more than $1.00.
Funding Sources:
For this pilot program, the military services reimbursed carriers and
forwarders for shipment-related costs from military personnel accounts.
These accounts were also used to fund move manager expenses. Any claims
that might have been filed with the government would have been funded
from the military services‘ operations and maintenance accounts.
Navy‘s Service Member Arranged Move Pilot Program:
Sponsored by the Navy, the Service Member Arranged Move Pilot Program
included only domestic outbound intrastate and interstate shipments for
Navy personnel moving from its installations located at Puget Sound,
Washington; San Diego, California; Norfolk, Virginia; New London,
Connecticut; and Whidbey Island, Washington. One of this program‘s
objectives was to offer Navy military personnel a set of moving choices
to meet their specific needs. This pilot program was one of three
choices offered. Military personnel moving from the above locations
could choose to move under the current personal property program, move
their own household goods, or participate in the pilot program. The
pilot program was initiated in April 1997 and began operations in
January 1998. Because the Navy decided not to scope the Service Member
Arranged Move Pilot Program comparable to other pilot programs (i.e.,
operational at multiple military services) and the pilot program did
not provide data as outlined by the Transportation Command‘s evaluation
plan, its inclusion in the Transportation Command‘s evaluation was
limited to a qualitative assessment.
Loss and Damage Claims:
Reimbursement for loss and damage claims was increased from depreciated
value to full replacement value, with the dollar amounts per move
increasing from $40,000 under the current program to $72,000 under the
Service Member Arranged Move Pilot Program. Additionally, the pilot
program provided direct claims settlement between military personnel
and their transportation providers and a requirement that
transportation providers settle claims within 60 days of receiving
claims forms from military personnel.
Counseling and Arranging Shipment Services:
Like the current program, the Service Member Arranged Move Pilot
Program also relied on personnel in the personal property shipping
offices to provide counseling services and arrange shipment and storage
of household goods and unaccompanied baggage. However, the shipping
office personnel at the origin installations participating in this
pilot program served as the single point of contact coordinating the
service members‘ moves and remained available throughout the move to
handle all issues, including claims. Unlike those participating in the
current program and other pilot programs, service members participating
in this pilot program identified the transportation provider they
desired to handle their household goods shipments after they completed
their reviews of participating providers‘ vendor quality books
(containing provider information and marketing materials) and of
surveys completed by previous pilot program participants. The personal
property office coordinator assigned to the service member had to
concur with the member‘s request, and the coordinator made actual
arrangements with the carrier.
Screening and Shipment Distribution Process for Transportation
Providers:
Staff in the program management office and personal property shipping
offices participating in the Service Member Arranged Move Pilot Program
initially screened transportation providers that wished to participate
in the pilot program based on providers‘ performance rather than low
cost. Letters of agreement were adopted to streamline the contracting
process and improve the quality of the move for Navy personnel.
According to pilot program officials, these letters of agreement
provided commercial best practices and enabled lessons learned from
prior pilot program efforts and industry to be incorporated into the
Navy pilot program. Actual contract awards were made on a case-by-case
basis based on the best value decision for each move. Transportation
providers used commercial tariffs in developing their bids for each
move.
Transportation providers approved to participate in the pilot program
submitted their bids for various origin and destination channels using
commercial tariffs. Bids were rejected if they did not fall within
acceptable percentage discounts. Feedback was provided monthly to
transportation providers, and those that became poor performers were no
longer offered household goods and unaccompanied baggage shipments.
Subsequently, service members who were planning their upcoming moves
relied on information contained in a nine-question survey that other
service members had completed after their moves and claims process
ended. Service members who volunteered to participate in this pilot
program had to manually review carrier books, which included documents
provided by the carriers and prior surveys completed by service members
who had been moved by the carriers. According to pilot program
officials, six carriers were terminated or canceled from the pilot
program-four for providing poor service and two for price gouging.
Visibility of Shipments during Relocation Process:
The Service Member Arranged Move Pilot Program relied upon the shipping
office personnel, who served as the single point of contact
coordinating the service members‘ moves, to maintain visibility of
shipments during the relocation process. In addition, the pilot program
provided both the personal property shipping office‘s and
transportation provider‘s toll-free numbers, as well as a pager to
service members to enhance the members‘ visibility of their shipments
during the relocation process.
Availability and Reliability of Data on Household Goods Shipments:
The Service Member Arranged Move Pilot Program did not initially
develop an alternative data management system to capture data on
shipments and payment records. By the end of the pilot program, the
Navy had developed a database to capture shipment data; however, the
system was not fully implemented or evaluated.
Unlike the other pilot programs, the Navy pilot program used local
personal property program personnel rather than third parties to review
all invoices for payment. Navy personnel who participated in the pilot
program completed their own surveys, mailing the paper forms to their
respective personal property program offices.
Other Service Member Arranged Move Pilot Program Features:
The Service Member Arranged Move Pilot Program was designed to offer
all shipments to small businesses, to provide direct claims settlements
between Navy personnel and the transportation providers, to make faster
payments to transportation providers through government purchase cards,
and to establish a stronger commitment from transportation providers by
offering long-term contracts.
Funding Sources:
Like the current program, the Navy reimbursed carriers and forwarders
for shipment-related costs from its military personnel account.
Personal property shipment office expenses were funded from the Navy‘s
operations and maintenance account. While information on claims filed
with the government was not provided, under this pilot program such
expenses would also be funded from the operations and maintenance
account.
[End of section]
Appendix III: Comments from the Department of Defense:
DEPUTY UNDER SECRETARY OF DEFENSE FOR LOGISTICS AND MATERIEL READINESS
3500 DEFENSE PENTAGON WASHINGTON, DC 20301-3500:
MAR 25 2003:
Mr. William M. Solis:
Director, Defense Capabilities and Management U.S. General Accounting
Office:
Washington, D.C. 20548:
Dear Mr. Solis:
This is the Department of Defense (DoD) response to the GAO draft
report, GAO-03-367, ’DEFENSE TRANSPORTATATION: Monitoring Costs and
Benefits Needed While Implementing a New Program for Moving Household
Goods‘, dated February 24, 2003 (GAO Code 350161).
The Department has reviewed the subject GAO draft report and offers the
attached comments.
Sincerely,
Allen W. Beckett,
Principal Assistant:
Signed by Allen W. Beckett:
GAO-03-367/GAO CODE 350161:
’DEFENSE TRANSPORTATION: MONITORING COSTS AND BENEFITS NEEDED WHILE
IMPLEMENTING A NEW PROGRAM FOR MOVING HOUSEHOLD GOODS“:
DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:
RECOMMENDATION 1: The GAO recommended that the Secretary of Defense
direct the Commander, U.S. TRANSCOM, to initiate actions to implement
the three recommendations contained in DOD‘s report to Congress within
budget constraints.
(Page 21/Draft Report):
DoD RESPONSE: DoD concurs with this recommendation. DoD is currently
developing a plan to implement the recommendations. The Department
expects new program implementation beginning with electronic payment
process in 4QFY03, with remaining changes beginning in 1QFY05,
continuing through 1QFY06 assuming the Military Services receive the
additional funds needed to fund program enhancements.
RECOMMENDATION 2: The GAO recommended that the Secretary of Defense
direct the Commander, U.S. TRANSCOM, to provide the Military Services
and Congress additional information to quantify the risk associated
with achieving the 13 percent projected cost estimate before the claims
process and performance-based service contracts recommendations are
implemented to provide the Military Services with information needed
for budgeting purposes. (Page 21/Draft Report):
DoD RESPONSE: DoD partially concurs with this recommendation. The
Department continues to believe that the factors used in developing its
projected cost estimate for the claims process and performance-based
service contract recommendations are supportable and the new program
can be implemented within the projected 13 percent increase over
current program costs. The cost analysis was based on historical data
and assumptions only where data was not available to provide analytical
information. The GAO report emphasizes a lack of confidence in the
adjustment made to the averaged pilot costs for economies of scale/
program efficiencies. The 5 percent reduction to the averaged pilot
costs for economies of scale is not only reasonable, but also very
conservative. The shipment costs in the pilots were based on a new set
of business practices with only 3,000-5,000 shipments annually spread
among the participating transportation providers. The current and new
program encompasses approximately 600,000 shipments annually, about 200
times larger scale of operations than the pilots. The larger volume
resulting in a lower cost per unit concept is a standard and accepted
law of economics. While the GAO reports the basis of the 5 percent
reduction for economies of scale was not based on either historical
experience or data provided by DoD, they did not declare the
methodology flawed.
GAO concurs with the methodology used for estimating costs for the
pilot programs and for two of the three cost reductions used to
estimate the future program cost increase. We are confident in all
three. We have presented the methodology to several of the Military
Services. One
Military Service performed their own analysis and validated the 13
percent. The methodology evaluated the ’unproductive“ cost drivers
(shipment weight, small/large carrier mix, economies of scale) in the
pilots and compensated for them using very conservative estimates.
DoD does not see value added in providing the Military Services or
Congress a formal risk assessment but will continue to work with the
Military Services as execution progresses to make sure they have all
information required for budget purposes.
The Department non-concurs with the GAO finding that the cost estimate
for implementing technology improvements will be $7M. The Department
maintains that the cost to implement a new web-based system will fall
between $4-6M. We projected costs of $5 million for development and
implementation. Training costs and approximately two months of costs
for independent verification and validation (IV&V) testing are
estimated to be $500,000 each respectively.
RECOMMENDATION 3: The GAO recommended that the Secretary of Defense
direct the Commander, U.S. TRANSCOM, to monitor costs for all
recommendations during the implementation phase to ensure that the
proposed changes are being achieved within an acceptable and predefined
range. (Page 21-22/Draft Report):
DoD RESPONSE: DoD concurs with this recommendation. Rolling out the new
program will require monitoring of costs to determine if the moving
industry partners are submitting bids that will allow the Department to
enhance this program within the 13 percent cost increase. The
Department intends to establish program requirements that will focus
more on quality of services (weighted best value score: 70 percent
performance and 30 percent cost). To address the issue of monitoring
costs, the plan includes a process to conduct a rate reasonableness
analysis upon receipt of the rates. Those rates found to be outside the
range of reasonableness will be rejected, and the carries will be given
one opportunity to resubmit their rates-a function of controlling and
evaluating costs. Only rates determined to be reasonable will be
available for use in the program. For the first time, the Department
will have one database with all cost data elements required by the
Department‘s functional and financial communities for cost evaluation
at the shipment level. The establishment of metrics, target/benchmark
performance indicators, and methodology for data collection will be
included in the updated program plan of action and milestone document
that was not available at the time of the GAO audit.
RECOMMENDATION 4: The GAO recommended that the Secretary of Defense
direct the Commander, U.S. TRANSCOM, to assess the effects of the three
recommendations on the personal property program (to be carried out
after the summertime peak moving season once the recommendations have
been implemented) to determine whether the anticipated improvements in
the program are being achieved at a reasonable cost. (Page 22/Draft
Report):
DoD RESPONSE: DoD concurs with this recommendation. The Department will
have the information needed to determine if anticipated improvements
have been achieved on a continuing basis. A customer satisfaction
survey will be implemented to assess improvements in the program and
identify how well the carriers are performing. This approach allows the
relocating member to access carrier performance and report customer
satisfaction. Also, since the surveys will be completed upon shipment
delivery, timely performance data will be
available. Carrier performance ratings will be established quarterly,
with the exception of the peak season, when performance ratings with be
established monthly.
[End of section]
Related GAO Products:
Defense Transportation: Final Evaluation Plan Is Needed to Assess
Alternatives to the Current Personal Property Program. GAO/NSIAD-00-
217R. Washington, D.C.: September 27, 2000.
Defense Transportation: The Army‘s Hunter Pilot Project Is Inconclusive
but Provides Lessons Learned. GAO/NSIAD-99-129. Washington, D.C.: June
23, 1999.
Defense Transportation: Plan Needed for Evaluating the Navy Personal
Property Pilot. GAO/NSIAD-99-138. Washington, D.C.: June 23, 1999.
Defense Transportation: Efforts to Improve DOD‘s Personal Property
Program. GAO/T-NSIAD-99-106. Washington, D.C.: March 18, 1999.
Defense Transportation: The Army‘s Hunter Pilot Project to Outsource
Relocation Services. GAO/NSIAD-98-149. Washington, D.C.: June 10, 1998.
Defense Transportation: Reengineering the DOD Personal Property
Program. GAO/NSIAD-97-49. Washington, D.C.: November 27, 1996.
FOOTNOTES
[1] The Deputy Under Secretary of Defense (Logistics) tasked the U.S.
Transportation Command to evaluate alternatives and develop
recommendations to enhance the department‘s current personal property
program.
[2] The three pilots included in the evaluation are the Military
Traffic Management Command‘s Reengineered Personal Property Program,
the Department of Defense‘s Full Service Moving Project, and the Navy‘s
Service Member Arranged Move Pilot Program.
[3] U.S. General Accounting Office, Defense Transportation: Efforts to
Improve DOD‘s Personal Property Program, GAO/T-NSIAD-99-106
(Washington, D.C.: Mar. 18, 1999).
[4] U.S. General Accounting Office, Defense Transportation: Final
Evaluation Plan Is Needed to Assess Alternatives to the Current
Personal Property Program, GAO/NSIAD-00-217R (Washington, D.C.: Sept.
27, 2000).
[5] DOD reported that the 146 day recovery time is due to the fact that
all current DOD contracts for shipment and storage of household goods
give the carriers 120 days from receipt of the government‘s demand to
pay, deny, or make a final written offer on the claims. While many
claims are settled in less that 120 days, mailing time and negotiations
to resolve disputes result in a slightly higher average settlement
time.
[6] For example, a 10,000-pound shipment would have a maximum carrier
liability for loss and damage of $12,500. Service members can make
shipments exceeding their weight allowance by paying the extra shipping
and storage costs for the overage in weight. However, the government‘s
liability is limited to $40,000 per move regardless of the excess
weight shipped and stored.
[7] U.S. General Accounting Office, Defense Transportation: The Army‘s
Hunter Pilot Project Is Inconclusive but Provides Lessons Learned, GAO/
NSIAD-99-129 (Washington, D.C.: June 23, 1999).
[8] In later discussions, department officials stated that ’economies
of scale“ should be changed to ’program efficiencies“ to reflect a more
efficient program with timely and accurate management data; member
counseling; and reduced loss, storage, and indirect costs.
[9] We believe that the methodology and data used to develop these cost
estimates are sound.
[10] GAO/NSIAD-99-129.
[11] U.S. General Accounting Office, Developing and Using
Questionnaires, GAO/PEMD-10.1.7 (Washington, D.C.: Oct. 1993) and U.S.
General Accounting Office, Using Structured Interviewing Techniques,
GAO/PEMD-10.1.5 (Washington, D.C.: June 1991).
[12] Our prior reviews focused on the department‘s efforts (1) to
develop a methodologically sound evaluation plan and (2) to collect
data according to the plan for future analysis and development of
recommendations for an improved departmentwide program.
GAO‘s Mission:
The General Accounting Office, the investigative arm of Congress,
exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability
of the federal government for the American people. GAO examines the use
of public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO‘s commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through the Internet. GAO‘s Web site ( www.gao.gov ) contains
abstracts and full-text files of current reports and testimony and an
expanding archive of older products. The Web site features a search
engine to help you locate documents using key words and phrases. You
can print these documents in their entirety, including charts and other
graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ’Today‘s Reports,“ on its
Web site daily. The list contains links to the full-text document
files. To have GAO e-mail this list to you every afternoon, go to
www.gao.gov and select ’Subscribe to daily E-mail alert for newly
released products“ under the GAO Reports heading.
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. General Accounting Office
441 G Street NW,
Room LM Washington,
D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470:
Public Affairs:
Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.
General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.
20548:
